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ABSTRACT 


The  quality  of  software  management  in  a  development  program  is  a  major  factor 
in  determining  the  success  of  a  program.  The  four  main  areas  where  a  software  program 
manager  can  affect  the  outcome  of  a  program  are  requirements  management, 
estimation/planning  management,  people  management,  and  risk  management.  In  this 
thesis  a  quality  management  metric  (QMM)  was  used  to  measure  the  performance  of  ten 
software  managers  on  Department  of  Defense  (DoD)  software  development  programs. 
Informal  verification  and  validation  of  the  metric  compared  the  QMM  score  to  an  overall 
program  success  score  for  the  entire  program  and  yielded  positive  correlation.  The 
results  of  applying  the  QMM  can  be  used  to  characterize  the  quality  of  software 
management  and  can  serve  as  a  template  to  improve  software  management  performance. 
Future  work  includes  further  refining  the  QMM,  applying  the  QMM  scores  to  provide 
feedback  and  appropriate  training  to  program  managers,  and  using  the  QMM  scores  as  an 
input  to  program  cost  and  schedule  estimation  methodologies  to  provide  better  program 
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EXECUTIVE  SUMMARY 


The  quality  of  management  in  a  software-development  program  is  a  major  factor 
in  determining  the  success  of  the  program.  Software  development  by  its  very  nature  can 
be  a  complex  process  because  the  product  or  products  developed  are  logical,  rather  than 
physical  systems,  and  therefore  the  development  can  not  be  approached  as  a 
manufacturing  problem.  Software  is  not  a  visible  and  tangible  product.  The  success  or 
failure  of  a  software-development  program  depends  to  some  degree  on  the  ability  of  the 
program  manager  to  estimate  and  plan  the  program,  coordinate  the  work  force  efforts, 
facilitate  communications,  define  requirements,  and  control  the  amount  of  risk  exposure 
the  program  experiences. 

Program-management  tools  have  been  developed  to  assist  the  program  manager  in 
estimating  the  cost  and  schedule  of  software  programs.  However,  the  estimation  tools 
available  assume  consistent  and  high-quality  program  management.  Although  the  quality 
of  software  development  management  tools  has  improved  over  the  past  thirty  years, 
software-development  programs  are  still  often  notoriously  behind  schedule  and  over 
budget.  Hence  there  still  appears  to  be  something  missing  in  the  software-program 
management  equation  that  has  been  overlooked  in  the  quest  to  produce  cost-effective, 
high-quality  software.  The  missing  something  may  be  good  management.  If  the  quality 
of  the  software  program  management  were  measurable  and  available  to  be  input  into  the 
costing  and  scheduling  tools,  then  the  resulting  estimates  could  be  provided  to  program 


xm 


managers  so  that  they  could  pinpoint  areas  of  software-program  management  in  which 
improvement  needs  to  be  made. 

The  Quality  Management  Metric  (QMM)  is  a  survey  instrument  which  measures 
in  a  repeatable  manner,  the  quality  of  management  for  a  software-development  program. 
The  QMM  consists  of  four  separate  metrics  which  measure  the  program  manager’s 
performance  in  the  areas  of  requirements  management,  estimation/planning  management, 
people  management,  and  risk  management.  The  results  of  the  four  metrics  are  then 
combined  to  provide  a  Quality  Management  Metric  score  which  predicts  the  quality  of 
the  program  management  and  the  success  of  the  program.  The  QMM  survey  instrument 
has  been  applied  to  measure  the  performance  of  three  software  managers  on  Department 
of  Defense  (DoD)  software  development  programs.  The  resulting  QMM  scores  of  the 
program  managers  were  informally  verified  and  validated  in  a  comparison  with  an  overall 
program  success  score  and  yielded  positive  correlation. 

This  thesis  evaluates  the  QMM  as  a  viable  software-management-quality  survey 
instrument,  for  monitoring,  evaluating,  and  improving  the  enactment  of  the  software 
process.  The  survey  and  this  research  are  a  step  in  that  direction. 

In  the  current  research,  the  QMM  survey  instrument  is  used  to  measure  the 
performance  of  ten  program  managers  on  DoD  software-development  programs.  In  order 
to  gain  a  measure  of  the  validity  of  the  QMM,  the  QMM  survey  instrument  was  given  to 
the  program  manager  and  also  to  one  or  two  other  independent  individuals  who  were 
closely  associated  with  the  program  and  knowledgeable  about  the  overall  practices  and 
success  of  the  program.  The  program  manager  QMM  scores  were  compared  to  the 
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program  manager’s  subjective  overall  program  success  scores  and  the  correlation  yielded 
strong  positive  results.  In  addition  to  the  comparison  of  the  QMM  score  with  the  overall 
program  scores,  the  results  from  the  four  separate  metrics  were  analyzed.  The  study 
examined  the  correlations  between  the  program  manager  scores  and  each  sub-area  metric 
and  the  QMM  scores  and  each  sub-area  metric  to  determine  whether  there  were 
relationships  between  the  overall  program  manager  performance  and  the  program 
manager  performance  in  a  specific  area  or  areas.  The  independent  QMM  scores  were 
compared  with  the  independent  overall  program  scores  and  yielded  a  strong  positive 
correlation.  The  independent  scores  in  each  of  the  four  sub-areas  were  also  compared 
against  the  independent  QMM  and  overall  program  scores  to  determine  whether  there 
were  any  relationships  between  the  overall  program  manager  performance  as  rated  by  the 
independent  and  the  overall  program  success,  as  rated  by  the  independent. 


From  the  results  of  the  data  analysis  several  conclusions  are  drawn: 

•  The  QMM  survey  instrument  yields  a  strong  correlation  with  the  QMM  score  and 
the  overall  program  score  for  both  the  program  manager  and  the  independent  data 
sets.  This  indicates  the  QMM  survey  instrument  does  measure  the  quality  of 
management  on  a  software-development  program  where  the  management  policies 
and  procedures  are  the  same  as  or  similar  to  the  currently  accepted  software 
engineering  management  practices,  as  defined  by  the  QMM  survey  instrument.  In 
the  case  where  the  management  practices  and  procedures  are  known  to  diverge 
from  the  currently  accepted  practices,  the  QMM  survey  instrument  may  be  used 
to  measure  the  level  to  which  the  currently  accepted  practices  are  implemented, 
but  will  likely  yield  a  lower  QMM  score  and  probably  predict  a  lower  success  rate 
for  the  program  than  the  actual  rate. 

•  The  data  indicates  that,  in  general,  program  managers  believe  that  they  are  good 
managers  of  people.  However,  the  independent  evaluators  do  not  believe  the 
program  managers  are  as  good  at  managing  people  as  the  program  managers 
themselves. 
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•  The  QMM  survey  instrument  may  be  useful  in  detecting  discrepancies  between 
the  program  manager’s  and  the  individual  team  member’s  perspectives  and 
understanding. 

•  The  results  of  the  study  indicate  that  there  is  a  great  deal  of  uncertainty  involved 
in  managing  estimation/planning  and  risk,  and  that  the  program  managers  who  are 
good  at  managing  the  uncertainty  in  one  of  these  areas  are  usually  good  at 
managing  the  uncertainty  in  the  other  area  as  well. 

•  The  data  correlations  between  the  program  manager  data  set  and  the  independent 
data  set  indicate  that  perhaps  the  area  of  requirements  management  is  more  visible 
and  better  understood  by  the  entire  development  team  than  other  areas  of 
management. 

Recommendations  for  future  work  include  updating  and  further  testing  the  QMM 
survey  instrument,  developing  a  mechanism  to  provide  the  program  managers  with 
feedback  on  their  management  performance  and  possible  courses  of  action  to  take  to 
improve  their  performance,  and  integrating  the  QMM  survey  instrument  results  into 
existing  cost,  schedule  and  risk  models  to  improve  program  estimation  accuracy. 
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I.  INTRODUCTION  AND  BACKGROUND 


A.  SOFTWARE  DEVELOPMENT 

In  addition  to  the  normal  development  difficulties  of  a  non-software  product 

development,  software  development  introduces  additional  complexity  and  potential 
problems  which  with  the  program  manager  must  deal.  Software  is  a  logical,  rather  than  a 
physical  system,  therefore,  development  cannot  be  approached  as  a  manufacturing 
problem.  Software  is  not  a  visible  and  tangible  product.  Per  dollar  spent,  software 
products  are  more  complex  than  other  engineering  products  [Ref.  1].  Software  is  flexible 
and  can  be  easily  changed,  at  least  on  the  surface.  Often  one  of  the  difficulties  facing  a 
program  manager  is  the  idea  that  software  is  flexible  and  easily  changed  at  any  point  in 
the  development  process.  Studies  have  shown  that  changing  software  requirements  late 
in  a  development  effort  has  a  much  more  significant  and  adverse  impact  on  the  program 
cost  and  schedule  than  changing  the  requirements  early  on  in  the  program.  However, 
since  the  development  effort  is  for  a  software  product  the  customer  often  asks  for  changes 
in  the  requirements  up  until  the  end  of  the  program.  The  program  manager  must  control 
the  amount  of  change  to  the  requirements  and  communicate  the  impact  of  proposed 
changes  to  the  customer  to  coordinate  and  maintain  control  of  the  change  process  and 
keep  the  program  on  schedule,  within  budget  and  produce  a  quality  product  the  customer 
enjoys. 

The  program  manager  is  responsible  for  gathering  the  requirements  and  ensuring 
they  fulfill  the  customer’s  needs  and  wants.  Getting  the  correct  requirements  is  difficult 
because  the  customer  often  does  not  know  exactly  what  they  want  until  they  have  seen 
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the  software  product  or  a  prototype  of  the  product  and  try  to  use  it.  Many  requirements 
are  implicit  in  the  customer’s  understanding  and  not  made  explicit  until  the  customer 
realizes  they  are  missing  during  the  use  of  the  product.  There  are  also  misinterpretations 
between  the  way  the  customers  expect  the  requirements  to  be  implemented  and  the  way 
the  developers  interpret  them. 

The  software  development  personnel  the  program  manager  leads  are  intelligent, 
educated  professionals  who  demand  good  management  to  be  highly  productive.  If  they 
believe  they  are  being  mismanaged  they  may  rebel  against  the  system.  The  developers 
thrive  on  challenging  tasks  which  are  perceived  as  being  significant,  appropriate  feedback 
and  peer  recognition,  and  require  a  good  work  environment  to  be  their  most  productive. 

The  software  development  process  is  further  complicated  by  the  interactions  and 
communication  required  for  a  successful  software  development  program.  The  program 
manager  is  responsible  for  facilitating  the  flow  of  communication  not  only  within  the 
development  program  but  of  equal  importance,  outside  of  the  program  with  the 
customers.  The  customers  must  be  kept  up  to  date  on  the  program  requirements,  cost, 
schedule,  problems,  and  possible  tradeoffs  so  they  can  provide  feedback  to  the  program 
manager  allowing  the  manager  to  make  informed  decisions  which  will  lead  to  a 
successful  program  product  and  a  satisfied  customer.  In  addition,  the  program  manager 
must  ensure  good  communication  between  the  developers  to  reduce  misunderstandings 
and  the  need  for  rework,  particularly  if  part  of  the  product  is  being  developed  by  one  or 
more  subcontractors. 
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Various  program  management  tools  have  been  developed  to  assist  the  program 
manager  in  estimating  the  cost  and  schedule  of  software  programs.  The  Constructive 
COst  Model  [Ref.  2]  or  COCOMO  model  is  a  well  known  software  development  cost  and 
schedule  estimation  tool.  It  is  representative  of  the  type  of  estimation  tools  available  to 
software  program  managers  in  that  it  uses  software  product  metric  measures  such  as  the 
estimated  number  lines  of  code  delivered  in  the  final  product  as  the  basis  for  the  program 
cost  and  schedule  estimations  throughout  the  program  development.  Although  the 
quality  of  the  software  development  management  tools  has  improved  over  the  past  thirty 
years,  software  development  programs  are  still  notoriously  behind  schedule,  over  budget, 
etc.  For  example,  in  March  1999  Constance  Fabian-Isaacs  and  Ed  Robinson  [Ref.  3] 
wrote. 


Software  development  projects  are  notorious  for  running  over  budget  and 
behind  schedule.  The  Center  for  Project  Management  in  San  Ramon, 
California  reports  that  99%  of  commercial  software  products  are  not 
completed  on  time,  within  budget,  or  according  to  specifications,  and  that 
the  average  project  is  underestimated  by  285%.  With  statistics  like  these, 
it’s  no  wonder  software  project  managers  are  looking  for  ways  to  improve 
their  processes. 


Clearly,  there  is  still  something  missing  in  the  software  program  development 
equation  which  has  been  overlooked  in  the  quest  to  produce  cost  effective,  high  quality 
software.  Randall  Jensen  suggests  the  missing  something  may  be  good  management 
which  he  states  is  “the  most  important  productivity  and  quality  driver”  [Ref.  4].  In  1981 
Barry  Boehm  [Ref.  2]  wrote. 

Poor  management  can  increase  software  costs  more  rapidly  than  any  other 
factor. 
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Gerald  Weinberg  took  Barry  Boehm’s  concept  one  step  further  by  summarizing 
the  four  most  important  cost  drivers  and  the  relative  importance  of  each  as  identified  by 
Barry  Boehm;  see  Figure  1. 
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Figure  1.  From  [Ref.  5]  The  Four  Most  Important  Cost  Drivers  on  a  Software 

Development  Program. 


From  Figure  1  it  is  obvious  that  the  quality  of  the  program  management  may  be  a 
major  factor  in  the  success  or  failure  of  a  software  development  program.  The 
COCOMO  cost  and  schedule  estimation  model,  and  many  others  like  it,  assume  uniform, 
good  management  for  both  the  developer  and  the  customer  organizations.  From  the 
frequency  with  which  software  development  programs  experience  cost,  schedule  and 
performance  overruns  and  problems,  the  assumption  of  good  management  may  be 
incorrect  in  many  cases.  If  the  quality  of  the  software  program  management  were 
measurable  and  available  to  be  input  into  the  cost  and  schedule  tools,  the  resulting 
estimates  would  provide  more  accurate  information  for  the  program  managers  and 
possibly  improve  the  overall  program  performance. 
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B.  INSTRUMENTS  FOR  MEASURING  THE  QUALITY  OF  SOFTWARE 

MANAGEMENT 

Finding  an  instrument  to  measure  the  quality  of  software  management  is  difficult. 
There  are  many  instruments  available  to  measure  the  product  metrics  which  focus  on 
individual  aspects  of  the  software  item  (usually  volume)  under  development.  There  are 
also  many  instruments  available  to  measure  the  process  metrics  which  focus  on  the 
activities  involved  in  software  development.  Both  metrics  typify  most  program 
performance  evaluations  and  ignore  any  consideration  of  the  quality  of  software 
management. 

The  People  Capability  Maturity  Model  (P-CMM)  [Ref.  6]  provides  a  framework 
from  which  the  program  organization  can  stabilize  their  procedures  and  get  repeatable 
results.  However,  the  P-CMM  does  not  provide  a  specific  set  of  metrics  for  measuring 
the  quality  of  or  the  improvement  of  program  management.  Instead,  the  P- 
CMM  assists  the  program  organization  in  defining  its  approach  to  measuring  the 
performance  of  the  management,  hiring  managers  which  conform  to  the  expected 
performance  and  training  its  managers  to  attain  the  desired  performance. 

There  are  few  if  any  instruments  available  to  measure  the  quality  of  the  software 
management.  One  instrument  which  does  measure  the  quality  of  software  management  is 
the  Quality  Management  Metric  (QMM)  developed  by  Martin  Machniak  [Ref.  7].  The 
QMM  measures  the  software  program  management  quality  through  the  use  of  a  survey 
which  is  given  to  the  program  manager.  From  the  program  manager’s  responses  to  the 
pair-choice  and  yes/no  questions  a  profile  of  the  program  manager  is  built  up  in  the  areas 
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of  requirements  management,  estimation/planning  management,  people  management  and 
risk  management.  From  the  manager’s  score  in  each  of  the  four  areas  a  total  QMM  score 
is  computed  which  predicts  the  manager’s  current  and  future  performance  as  well  as  the 
program’s  current  and  future  performance. 

To  date  the  QMM  has  undergone  limited  testing.  Martin  Machniak  used  the 
QMM  survey  instrument  to  measure  the  performance  of  three  software  program 
managers  on  Department  of  Defense  (DoD)  software  development  programs.  The 
resulting  QMM  scores  of  the  program  managers  were  informally  verified  and  validated  in 
a  comparison  with  an  overall  program  success  score  and  yielded  positive  correlation 
[Ref.  7], 

This  thesis  evaluates  the  QMM  as  a  viable  software  management  quality  survey 
instrument.  The  long-term  goal  is  to  provide  a  means  for  monitoring,  evaluating,  and 
improving  the  enactment  of  the  software  process.  The  survey  and  this  research  are  a  step 
in  that  direction. 

C.  SCOPE  OF  THE  RESEARCH 

Since  the  QMM  has  been  tested  in  a  limited  fashion  (3  programs),  the  thesis 

further  explores  the  validity  of  the  QMM  in  measuring  the  quality  of  software  managers 

and  in  predicting  the  success  of  programs.  The  QMM  survey  instrument  was  used  to 

measure  the  performance  of  ten  program  managers  on  DoD  software  development 

programs.  To  gain  a  measure  of  the  validity  of  the  QMM,  the  QMM  survey  instrument 

was  given  to  the  program  manager  and  also  to  one  to  two  other  individuals  who  were 

closely  associated  with  the  program  and  knowledgeable  about  the  overall  practices  and 
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success  of  the  program.  The  program  manager  QMM  scores  were  compared  to  the 
program  manager  overall  program  success  scores  and  the  correlation  yielded  positive 
results.  In  addition  to  the  comparison  of  the  QMM  score  with  the  overall  program  scores, 
the  results  from  the  four  sub-areas  of  the  QMM:  requirements  management, 
estimation/planning  management,  people  management,  and  risk  management  were 
analyzed  as  well.  The  study  examined  the  correlations  between  the  program  manager 
scores  and  each  sub-area  and  the  QMM  scores  and  each  sub-area  to  determine  if  there 
were  any  relationships  between  overall  program  manager  performance  and  the  program 
manager  performance  in  a  specific  area  or  areas.  The  independent  QMM  scores  were 
compared  with  the  independent  overall  program  scores  and  yielded  a  positive  correlation. 
The  independent  scores  in  the  four  sub-areas  were  also  compared  against  the  independent 
QMM  and  overall  program  scores  to  determine  if  there  were  any  relationships  between 
the  overall  program  manager  performance  as  rated  by  the  independent  and  the  overall 
program  success  as  rated  by  the  independent. 

From  the  results  of  the  data  analysis  several  conclusions  are  drawn,  these  are 
presented  in  Chapter  V.  Recommendations  for  future  work  are  also  presented  in  Chapter 
V. 
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II.  RELATED  WORK 


A.  BACKGROUND  OF  THE  QMM 

The  QMM  is  a  repeatable  set  of  metrics  that  can  be  used  to  assess  the  quality  of 

software-development  management.  It  can  also  be  used  to  measure  improvement  in  the 
quality  of  the  management  of  software  development  and  predict  future  success  levels  of 
software-development  projects.  The  definition  of  software  development  extends  to  both 
development  of  new  software  and  maintenance  efforts  to  maintain  and  extend  the  existing 
functionality  of  systems.  Application  of  the  QMM  outside  of  the  software-development 
domain  has  not  been  addressed  in  the  development  or  testing  of  the  QMM  survey 
instrument. 

The  QMM  survey  instrument  consists  of  four  categories  of  software-management 
metrics:  requirements  management,  estimation/planning  management,  people 
management,  and  risk  management.  The  scores  for  each  category  provide  an  indication 
of  the  quality  of  the  program  manager’s  management  of  a  software-development  project. 

The  program  manager  is  asked  to  fill  out  the  QMM  survey  instrument  to  attain  a 
measure  of  the  quality  of  the  management  on  the  software  development  program.  The 
resulting  QMM  score  predicts  the  program  manager’s  overall  performance  and  the 
success  of  the  program.  Each  of  the  four  sections  of  the  QMM  has  the  same  number  of 
total  points  possible.  The  total  points  for  each  section  are  multiplied  by  its  relative 
Importance  Coefficient  (IC)  to  yield  a  weighted  score.  After  the  weighted  scores  for  each 
section  are  computed,  they  are  summed  together  to  yield  the  Quality  Management  Metric 
(QMM)  score  [Ref.  7]. 
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The  IC  for  each  section  was  determined  from  the  relative  rankings  of  importance 
of  each  of  the  sections.  The  IC  rankings  were  determined  through  interviews  with  senior 
program  managers  and  focus  groups. 

The  overall  QMM  score  is  made  up  of  the  sum  of  the  scores  of  the  each  of  the 
individual  sections  multiplied  by  an  IC  for  that  section.  The  QMM  equation  is  as  follows 
[Ref.  7]: 

QMM  =  0.92  RqM  +  0.67  EPM  +  1.86  PM  +  0.55  RkM 

The  QMM  is  the  sum  of  the  four  components: 

RqM  is  the  requirements  management  metric 

EPM  is  the  estimation/planning  metric 

PM  is  the  people  management  metric 

RkM  is  the  risk  management  metric 


The  QMM  survey  instrument  is  intended  to  pertain  to  only  one  program  at  a 
specific  point  in  time.  Each  program  manager  who  participates  in  the  survey  is  asked  to 
answer  the  survey  questions  from  the  perspective  of  what  was  happening  on  a  specific 
program  at  a  specific  point  in  time,  such  as  a  milestone  or  delivery. 

It  is  envisioned  that  the  QMM  survey  instrument  can  be  used  to  measure  the 
improvement  in  a  program  manager’s  performance.  The  program  manager  can  take  the 
survey  at  different  points  in  time  on  the  same  program  or  on  two  different  programs,  and 
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the  results  of  the  surveys  can  be  compared  to  obtain  an  indication  of  the  improvement  of 
the  program  manager  in  managing  the  software-development  program. 

The  questions  used  on  the  QMM  survey  instrument  were  gathered  from 
periodicals,  textbooks,  interviews  with  senior  program  managers,  and  focus  groups.  The 
intent  was  to  be  as  inclusive  as  possible  with  respect  to  judging  the  quality  of 
management.  Appendix  A  contains  the  QMM  survey  instrument  forms  with  the 
weighting  for  the  individual  questions. 

Part  one  of  the  questionnaire  contains  pair-choice  questions.  These  questions 
require  the  participant  to  choose  one  of  two  statements  that  best  describes  what  was 
happening  on  the  program  at  the  evaluation  point  in  time.  The  choice  does  not  have  to 
match  exactly:  the  participant  selects  the  choice  which  best  describes  the  program.  The 
model  for  this  type  of  survey  instrument  is  the  Myers-Briggs  Type  Indicator 
questionnaire.  Each  pair  choice  represents  two  differing  ideas  in  an  effort  to  ascertain  a 
tendency  of  the  individual.  Often  the  pair  choices  are  repeated  with  different  wording  to 
confirm  earlier  choices  and  measure  the  strength  of  the  tendency  [Ref.  7],  Each  section 
has  approximately  70  questions  and  each  section  has  a  total  of  70  points  for  part  one  of 
the  questionnaire. 

Part  two  of  the  questionnaire  contains  questions  to  be  answered  with  yes,  no,  or 
not  applicable(n/a).  This  part  of  the  questionnaire  is  user-friendly  for  conducting  surveys, 
requiring  minimum  writing.  Each  question  has  a  point  value  based  upon  the  relative 
importance  of  the  question  in  that  section.  Each  section  has  approximately  fifty  questions 
and  a  maximum  score  of  sixty-two  points.  The  entire  survey,  including  both  parts  for  all 
four  sections  contains  457  questions. 
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The  QMM  survey  instrument  was  conceived  and  targeted  for  high-level  program 
managers.  High-level  program  managers  are  often  in  charge  of  more  than  just  a  software 
development  effort,  including  system-level  design,  hardware  development,  integration 
and  testing  of  an  entire  system  which  includes  software  development  as  a  component  of 
the  overall  system.  The  high-level  program  manager  is  asked  to  answer  the  questions  as 
the  questions  pertain  to  the  software  development  portion  of  the  program  in  question. 

In  order  for  a  program  to  qualify  for  participation  in  a  QMM  survey,  the 
minimum  requirement  is  that  a  program  manager  be  in  charge  of  at  least  three  software 
developers.  The  software  developers  do  not  have  to  be  working  as  software  coders;  they 
can  be  working  in  any  of  the  software  development  areas  such  as  design,  requirements 
analysis,  documentation,  coding,  and  testing.  A  requirement  of  the  software  developers 
is  that  they  have  defined  areas  of  responsibility.  These  areas  can  be  shared  with  other 
developers,  but  specific  developers  are  assigned  to  each  area. 

Administration  of  the  survey  was  conducted  such  that  the  participants  had  no 
information  on  the  point  totals  or  relative  weightings  of  the  questions  before  hand.  This 
was  done  to  avoid  introducing  additional  bias  into  the  results  of  the  survey.  Each  survey 
was  given  to  the  participant  in  a  private,  one-to-two  hour  interview.  The  results  were 
manually  scored  to  focus  attention  on  the  entire  process  and  de-emphasize  focusing  on 
only  the  final  Quality  Management  Metric  (QMM)  score  [Ref.  7].  In  order  to  obtain 
complete  and  full  disclosure  on  the  surveys,  the  results  were  reported  anonymously  as 
programs  A,  B,  and  C. 

Collectively,  the  measures  in  the  four  sections  give  an  objective  view  of  the 
quality  of  the  software  management.  Thus,  two  programs  scoring  equally  on  product  and 
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process  metrics  can  be  further  measured  and  compared  on  the  basis  of  the  quality  of  their 
management. 

B.  SOFTWARE  MANAGEMENT  COMPONENTS  OF  THE  QMM 

The  QMM  provides  a  repeatable  set  of  metrics  to  determine  the  quality  of 

software  management,  measure  the  improvement,  and  predict  future  success  levels  of 
programs.  The  QMM  survey  instrument  is  made  up  of  four  sections:  requirements 
management,  estimation/planning  management,  people  management,  and  risk 
management. 

1.  Requirements  Management 

Requirements  management  focuses  on  managing  the  process  of  extracting, 
developing,  defining,  and  refining  the  requirements  of  a  software  development  program 
[Ref.  7].  The  QMM  requirements  metric  is  not  a  product  or  process  metric  for 
requirements  nor  is  it  intended  to  be  one.  Many  product  and  process  metrics  already  exist 
for  requirements.  The  QMM  requirements  metric  is  intended  to  objectively  measure  the 
quality  of  the  program  manager  in  the  area  of  requirements  management. 

Requirements  are  necessary  for  software  programs  because  they  define  the 
product  to  be  produced.  Well  documented  and  written  requirements  unambiguously  state 
the  capabilities  which  the  software  product  will  have  upon  completion  and  form  a 
contract  between  the  customer  and  the  developer  which  defines  the  scope  of  the  work  to 
be  accomplished.  The  program  manager  is  responsible  for  establishing  procedures  to 
ensure  that  the  requirements  specification  is  complete,  consistent,  readable,  lacks 
ambiguity,  and  can  be  traced  to  origins.  The  program  manager  must  also  ensure 
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stakeholder  involvement  in  extracting  and  refining  the  requirements  to  make  sure  the 
software  product  will  satisfy  the  customer’s  needs. 

The  program  manager  is  responsible  for  validating,  prioritizing,  and  conducting 
tradeoff  studies  on  the  requirements.  Part  of  managing  the  customer  expectations  is 
validating  the  requirements.  The  requirements  should  be  scrubbed  to  be  certain  they 
reflect  the  customer  needs  accurately.  Once  the  requirements  are  validated,  they  should 
be  prioritized.  Every  software  development  program  has  cost  and  schedule  constraints. 
Often,  the  requirements  which  are  extracted  would  extend  the  development  program 
beyond  the  cost  and  schedule  constraints  and  so  the  requirements  are  prioritized  with  the 
customer  so  that  when  a  constraint  exists,  the  highest  priority  requirements  are  executed 
and  the  lower  priority  requirements  are  cut.  Requirements  prioritization  goes  hand-in- 
hand  with  requirements  tradeoff  studies.  The  customer  and  the  program  manager  need  to 
know  their  development  options  to  determine  the  appropriate  requirements  and  the 
priority  of  the  requirements  for  the  software  product. 

The  program  manager  is  responsible  for  implementing  change  control 
management  for  the  requirements.  Scope  creep,  where  the  requirements  expand  due  to 
new  or  changed  customer  requirements  or  through  the  developers  throwing  in  nice-to- 
have  features,  increase  the  complexity  of  the  product,  thereby  increasing  the  cost  to  test 
and  debug  the  product.  The  new  requirements  also  eat  into  the  cost  and  schedule 
constraints  of  the  program,  possibly  to  the  point  of  reducing  the  number  of  original 
requirements  which  can  be  accomplished.  Requirements  change  control  is  necessary  to 
limit  the  amount  of  unauthorized  change,  and  ideally,  to  assist  in  managing  the 
customer’s  expectations. 
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The  QMM  requirements  metric  questionnaire  evaluates  the  program  manager  on 
the  formality  of  the  requirements,  the  establishment  of  procedures,  extraction 
management,  change  management,  and  testability  of  requirements.  The  questionnaire 
does  not  seek  to  determine  the  quality  of  judgment  on  any  specific  decisions  made.  The 
thrust  of  the  questions  is  to  establish  the  structure,  if  any,  laid  out  by  the  program 
manager  in  the  area  of  requirements  [Ref.  7].  How  formal  are  the  processes?  How  well 
understood,  and  implemented  are  the  processes?  Who  is  involved  in  the  extraction 
process  and  when  is  it  done?  How  is  change  handled?  Is  each  requirement  testable  and 
when  is  testing  considered  during  the  development? 

2.  Estimation/Planning  Management 

Estimation  and  planning  management  focuses  on  establishing  procedures  and 
techniques  to  accurately  estimate  the  costs  and  schedule  of  the  development  effort,  and 
from  these  estimates  properly  plan  the  program.  Accurate  estimates  are  necessary  to 
produce  realistic  plans  which  are  essential  to  avoid  cost  overruns,  late  deliveries, 
inefficiency,  poor  reliability,  and  lack  of  user  acceptance. 

The  program  manager  is  responsible  for  establishing  a  frame  of  reference  for  the 
program  estimates  and  plans  as  well  as  an  ability  to  measure  against  the  reference  to 
determine  program  progress.  Three  types  of  measures  used  to  assess  program  progress 
are  as  follows:  product  measures,  process  measures,  and  resource  measures.  Product 
measures  are  based  upon  the  products  produced:  lines  of  code,  pages  of  documentation, 
and  number  of  modules  completed  within  each  phase  of  the  program.  Process  measures 
quantify  the  behavior,  strategy,  and  the  execution  of  the  processes  used  to  develop  the 
product.  Process  measures  are  based  on  such  things  as  the  number  of  defects  found  in 
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testing,  the  number  of  requirements  changes,  and  the  milestones  which  are  met. 
Resource  measures  are  based  on  the  number  of  labor  hours  expended,  the  appropriate 
assignments  of  personnel,  and  the  proper  work  environment  to  maximize  productivity. 

The  program  manager  is  responsible  for  implementing  the  proper  procedures  and 
structure  to  track  the  product,  process,  and  resource  metrics  throughout  the  program,  and 
ensuring  the  measures  being  used  will  yield  the  most  accurate  and  useful  results  for  the 
program. 

The  QMM  estimation/planning  metric  questionnaire  does  not  seek  to  advocate  a 
specific  estimation  technique  but  rather,  it  seeks  to  quantify  the  quality  of  the 
management  of  the  estimation  and  planning  process.  The  questions  determine  whether 
the  program  manager  is  performing  both  initial  and  follow  up  estimation  and  planning.  Is 
documentation  completed  and  used?  Are  currently  acceptable  software  engineering 
management  methods  and  practices  being  employed?  Is  the  program  manager  managing 
the  estimation  and  planning  process  sufficiently  to  give  confidence  to  the  program 
estimates  and  plans? 

3.  People  Management 

The  QMM  looks  at  people  management  from  a  specific  software 
development/management  perspective.  It  focuses  on  the  unique  qualities  and  needs  of 
people  working  in  a  software-development  environment. 

The  people-management  metric  focuses  on  the  program  manager’s  ability  to 
allocate  human  resources  appropriately,  implement  a  structure  to  facilitate 
communication  both  within  and  external  to  the  program,  and  provide  leadership  to  the 
program.  People  management  is  the  most  important  component  of  the  QMM.  Quality 
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people  management  increases  the  productivity  of  the  workforce  and  increases  the 
program’s  chance  to  meet  budget  allocations  and  schedules  and  be  successful. 

The  program  manager  is  responsible  for  establishing  the  organizational  culture  at 
the  program-development  level.  This  includes  recruiting,  training,  allocating  tasks, 
building  teams,  empowering  team  members,  implementing  good  organization,  mentoring, 
establishing  a  good  work  environment  and  reward  structure,  and  promoting  good 
communications.  The  developer’s  productivity  is  more  significantly  impacted  by  the 
team  work  environment  than  the  overall  organizational  work  environment.  The  program 
manager  is  responsible  for  developing  the  proper  work  environment  on  the  team  level  to 
promote  the  maximum  individual  productivity  from  team  members.  The  program 
manager  is  also  responsible  for  fostering  open  communications  within  the  development 
team  (horizontal),  with  the  program  manager  (vertical),  and  with  the  users  and 
stakeholders  (external)  to  the  extent  possible  within  the  overall  organizational  structure. 
Program  manager  leadership  and  mentoring  are  important  in  creating  and  maintaining  the 
team  work  environment  and  communication  infrastructure.  The  leadership  of  the 
program  manager  is  important  in  enabling  the  program  manager  to  establish  good 
practices  for  managing  human  resources. 

The  people-management  section  of  the  QMM  is  composed  of  four  sub-sections: 
human  resources,  leadership,  communication,  and  technical  competence.  The  human 
resources  section  covers  human-resource  management.  The  leadership  section  covers  the 
personal  leadership  skills  exhibited  and  the  leadership  mentoring  provided  by  the 
program  manager.  The  communication  section  covers  what  communication  protocols  are 
set  up  for  the  program  organization  and  used  individually  by  the  program  manager.  The 
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technical  competency  questions  cover  the  level  of  competency  of  the  program  manager 
within  the  program  on  a  technical  level. 

The  QMM  People-Management  metric  questionnaire  covers  the  people- 
management  skills  the  program  manager  exhibits  and  the  type  of  organizational  and 
communication  structure  the  program  manager  establishes  at  the  development-team  level. 
The  program  managers  are  rated  on  their  personal  leadership  skills  and  team  mentoring. 
The  organizational  structure  set  up  determines  if  the  program  manager  creates  an  adult- 
to-adult  or  master-servant  type  of  relationship  between  the  development  team  and  the 
manager.  The  adult-to-adult  management  style  is  one  in  which  the  manager  asks  for  the 
development  team’s  inputs  before  making  decisions  or  empowers  the  development  team 
to  make  decisions.  The  master-servant  type  of  management  style  is  one  in  which  the 
manager  does  not  ask  for  inputs  from  the  development  team;  rather,  the  manager  dictates 
the  decisions  and  the  tasks  to  the  team.  The  adult-to-adult  is  more  favorable  for 
increasing  team  member  job  satisfaction,  according  the  focus  group  feedback. 

The  questions  also  ascertain  the  communications  protocols  which  the  program 
managers  establish  for  program  communication  as  well  as  the  communications  formats 
which  the  program  managers  use  themselves. 

4.  Risk  Management 

Risk  management  is  the  process  of  identifying,  mitigating,  and  eliminating 
potential  problems  before  they  can  do  result  in  a  loss,  such  as  a  efficacy  (i.e., 
productivity)  loss. 

The  QMM  examines  risk  management  by  looking  at  the  components  of  risk 
management:  risk  assessment,  risk  control,  risk  communication,  and  risk  avoidance.  Risk 
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assessment  is  comprised  of  identifying  the  risks,  analyzing  the  risks,  and  prioritizing  the 
risks.  It  is  important  to  identify  each  risk  individually  so  that  they  can  be  dealt  with  on  an 
individual  basis.  The  risks  are  analyzed  for  their  estimated  impact  to  the  program  if  they 
occur,  and  then  are  prioritized.  The  prioritization  allows  the  program  manager  to 
concentrate  the  limited  program  resources  on  those  risks  which  are  the  most  likely  to 
occur  and  which  will  result  in  the  largest  of  the  losses  to  the  program  if  they  do  occur. 

Risk  control  is  comprised  of  risk  planning,  risk  tracking,  and  risk  resolution.  Risk 
control  consists  of  managing  the  risks  to  obtain  the  desired  outcomes.  Risk  planning 
involves  determining  appropriate  actions  to  mitigate  individual  risks,  prioritizing  the 
actions,  and  integrating  them  into  an  executable  risk  management  plan.  Risk  tracking 
involves  tracking  the  status  of  the  risks  and  their  mitigation  actions  through  metrics.  Risk 
resolution  is  the  execution  of  the  risk  mitigation  plans  for  each  risk. 

Risk  communication  is  the  process  of  exchanging  risk  information  throughout  the 
organization  at  all  levels.  This  activity  is  critical  to  the  management  of  risk,  allowing  the 
program  manager  to  maintain  a  level  of  awareness  of  the  risks,  their  impacts,  and  the 
success  or  failure  of  the  risk  mitigation  strategies. 

Risk  avoidance  is  one  possible  strategy  for  risk  resolution  which  involves 
avoiding  the  more  risky  development  strategies  or  technical  challenges.  Examples  of  risk 
avoidance  include  using  a  proven  compiler  and  development  environment  instead  of 
using  a  new  and  untried  compiler  and  development  environment,  and  deciding  to  use  a 
proven  technology  component  instead  of  relying  upon  the  development  of  a  new  cutting 
edge  technology. 
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All  programs  have  some  element  of  risk  as  do  all  areas  of  the  program — 
requirements  management,  estimation/planning  management,  and  people  management. 
Quality  management  acknowledges  the  presence  of  risk  and  implements  some  level  of 
systematic  risk  management. 

The  QMM  risk  management  metric  measures  the  risk  management  structure 
which  is  set  up  by  the  program  manager  to  deal  with  risks.  The  questionnaire  examines 
the  structures  used  by  the  program  management  for  identification,  monitoring,  and 
managing  risk.  The  questions  determine  whether  the  program  manager  has  implemented 
strategies  and  personnel  to  assess,  explore,  and  prioritize  all  reasonable  risks  and  also  if  a 
risk  management  plan  exists  and  is  implemented  and  updated  throughout  the  program’s 
development. 

C.  INITIAL  VALIDATION  AND  VERIFICATION  OF  THE  QMM 

The  approach  to  the  verification  and  validation  of  the  QMM  was  informal.  Three 

DoD  programs  were  evaluated  for  a  QMM  score.  The  program  manager  and  one 
program  development  team  member  evaluated  the  program  management  on  programs  A 
and  C.  On  program  B,  the  program  manager  and  two  program  development  team 
members  evaluated  the  program  management. 

In  order  to  provide  a  frame  of  reference  in  which  to  correlate  the  survey  results, 
two  other  measures  were  developed  and  used.  These  two  measures  are  the  QMM 
percentage  score  and  the  overall  program  success  score  [Ref.  7]. 

The  QMM  percentage  score  is  a  derived  measure  of  the  overall  QMM  score.  The 
maximum  possible  score  for  the  entire  survey  is  528  points  and  the  minimum  possible 
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score  is  -130.86  as  part-two  questionnaires  contain  negative  point  response  values.  In 
order  to  obtain  a  QMM  percentage  score,  the  following  steps  are  taken.  First,  the  survey 
minimum  possible  score  is  normalized  to  zero.  Since  the  survey  minimum  QMM  score  is 
-130.86,  130.86  is  added  to  the  survey  minimum  possible  score  so  that  it  equals  zero. 
Correspondingly,  130.86  must  be  added  to  both  the  survey  maximum  score  possible  and 
to  the  actual  QMM  score  obtained  in  the  survey.  Since  the  QMM  survey  maximum 
possible  score  is  528.0,  the  resulting  normalized  survey  maximum  possible  score  is 
658.86  [Ref.  7]. 

To  obtain  a  QMM  percentage  score,  the  normalized  QMM  score  obtained  from 
the  survey  is  divided  by  the  normalized  survey  maximum  possible  QMM  score  and  then 
multiplied  by  100.  The  equations  are  as  follows  [Ref.  7]: 

QMMmin  +  130.86  =  0.00  =  QMMmin  normalized 

QMMmax  +  130.86  =  658.86  =  QMMmax  normalized 

QMMscore  +  130.86  =  QMMscore  normalized 

(QMMscore  normalized/ QMMmax  normalized)  *  100  =  QMMpercentage  score 

The  overall  program  success  score  is  a  subjective  number  assigned  by  the  survey 
participant  rating  the  overall  success  of  the  program.  The  success  of  the  program  is 
measured  in  terms  of  how  well  the  final  product  performs  on  a  scale  of  zero  to  ten.  On 
this  scale  zero  corresponds  to  complete  and  utter  failure  with  no  viable  product  produced. 
A  score  of  ten  on  the  scale  is  not  realizable,  except  theoretically;  it  corresponds  to 
complete  success,  no  bugs  in  the  product,  delivery  on  time  and  within  budget,  and 
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complete  customer  satisfaction  with  the  end  product.  The  cause  for  success  or  failure  of 
the  program  is  not  important.  It  may  or  may  not  be  associated  with  any  actions  involving 
program  management. 

In  addition  to  the  QMM  percentage  score  and  the  overall  program  score,  a  mean 
overall  success  score  was  obtained  for  each  of  the  three  programs.  The  mean  overall 
success  score  is  derived  from  each  survey  participant’s  overall  program  success  score  and 
at  least  two  other  individuals  (mostly  users,  or  those  somehow  associated  with  the 
program  or  delivered  product)  able  to  judge  the  overall  success  of  the  program  [Ref.  7]. 

The  goal  underlying  the  data  analysis  was  to  determine  any  correlation  between 
the  participant’s  QMM  score,  their  individual  overall  program  score,  and  the  mean 
overall  success  ranking  of  the  program.  This  was  done  by  comparing  the  QMM 
percentage  score  with  the  individual  overall  program  success  score  and  the  mean  overall 
success  score  for  the  particular  program.  For  example,  an  overall  program  success  score 
of  seven  corresponding  to  a  QMM  percentage  score  of  seventy  percent  plus  or  minus  five 
percent  indicates  a  strong  correlation.  An  overall  program  success  score  corresponding 
to  a  QMM  percentage  score  of  between  plus  or  minus  five  and  fifteen  percentage  points 
of  the  QMM  percentage  score  indicates  fair  correlation.  An  overall  program  success 
score  corresponding  to  more  than  plus  or  minus  fifteen  percentage  points  of  the  QMM 
percentage  score  indicates  weak  correlation.  If  the  QMM  percentage  score  has  weak 
correlation  with  the  program  overall  success  score,  then  the  QMM  metric  is  still  valid  as 
programs  could  conceivably  fail  for  a  variety  of  reasons,  including  a  poor  choice  of 
technology  or  a  funding  shortfall.  The  reverse  condition  may  also  be  true  for  explaining 
successful  programs  with  low  quality  of  software  management.  However,  it  is  typically 
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expected  that  successful  software  programs  follow  superior  software  program 
management  [Ref.  7]. 

The  results  of  the  data  analysis  are  shown  in  Table  1  [Ref.  7]. 


Program 

Program  A 

Program  B 

Program  C 

Participant 

A-PM 

A  - 1 

B  - 1 

B-2 

C-PM 

C-l 

QMM  Score 

338 

106 

47 

198 

189 

QMM  Percent 

71.2% 

68.8% 

78.5% 

35.9% 

27.0% 

49.9% 

48.6% 

Success  Score 

7 

7 

9 

4 

3 

4 

4 

Mean  Success 
Score  (0  - 10) 

7 

4 

4 

Table  1.  From  [Ref.  7]  Results  Summary  Comparison. 


Program 

Program 

Score 

QMM 

Score 

Req. 

Man. 

Est ./ 
Planning 
Man. 

People 

Man. 

Risk 

Man. 

A-PM 

7 

71.0% 

67.1% 

73.7% 

80.0% 

41.0% 

A  - 1 

7 

68.7% 

65.2% 

62.3% 

80.0% 

41 .0% 

B-PM 

9 

78.4% 

66.5% 

76.6% 

85.6% 

72.9% 

B  - 1 

4 

35.8% 

41.3% 

24.6% 

34.4% 

36.1% 

B-2 

3 

26.9% 

28.4% 

24.0% 

27.5% 

15.1% 

C-PM 

4 

49.8% 

29.7% 

15.6% 

80.6% 

15.1% 

C-1 

4 

48.5% 

37.4% 

34.7% 

68.1% 

10.8% 

Table  2.  Data  From  the  QMM  Survey  Summary  Sheets 


The  QMM  summary  sheets  for  each  survey  are  summarized  in  Table  2.  The 

program  manager  for  program  A  is  referred  to  in  the  tables  as  A  -  PM,  the  independent 

evaluator  on  program  A  is  referred  to  as  A  -  1,  and  so  on.  An  error  in  the  math  on  the 

QMM  summary  sheets  has  been  corrected  in  the  results  presented  in  Table  2.  Therefore, 
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the  QMM  percentage  scores  vary  slightly  between  Table  1  and  Table  2.  Table  2  also 
presents  the  percentage  of  the  points  scored  in  each  of  the  four  sections:  requirements 
management,  estimation/planning  management,  people  management,  and  risk 
management. 

Programs  A  and  C  exhibit  correlation  between  the  QMM  percentage  score  and 
both  the  overall  program  success  score  and  the  mean  overall  success  score.  By 
examining  Table  2  it  can  be  seen  that  program  A  shows  a  weak  risk-management  section. 
This  appears  to  be  correct,  as  risk  management  for  this  program  was  not  emphasized. 
The  higher  scores  in  the  other  sections  reflect  the  fact  that  program  A  was  highly 
structured  and  planned,  had  successful  requirements  extraction,  and  enjoyed  good 
technical  success  with  their  deliverables. 

Program  C  had  essentially  no  risk  management,  little  planning,  and  poor 
requirements  extraction.  Once  again  this  is  reflected  in  the  scores  in  Table  2.  However, 
this  program  did  have  good  people-management  which  allowed  them  to  deliver  a  usable 
product,  the  high  people  management  skill  level  is  also  reflected  in  Table  2. 

Program  B  appears  to  have  a  disconnect  between  the  perceptions  of  the  program 
manager  and  the  development  team  members  interviewed.  The  difference  in  perception 
is  possibly  due  to  a  widely  documented  effect  in  social  psychology  in  which  people 
prefer  to  see  themselves  in  a  self-enhancing  fashion.  On  the  job,  approximately  ninety 
percent  of  managers  and  workers  rate  their  performance  superior  to  that  of  their  peers 
[Ref.  7], 

The  conclusion  of  the  thesis  was  that  the  goal  of  creating  repeatable  metrics  for 
determining  the  quality  of  software  management  was  obtained.  Further  work  is  needed  to 
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refine  the  survey  questions  in  wording  and  clarity,  the  point  values  for  the  questions  need 
to  be  refined  to  improve  the  survey,  and  additional  case  studies  are  required  to  fine  tune 
the  QMM  metrics. 


25 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


26 


III.  CURRENT  WORK  WITH  THE  QMM 


A.  METHODOLOGY  AND  PROTOCOL 

The  goal  of  the  current  work  with  the  QMM  is  to  further  explore  the  validity  of 

using  the  QMM  for  measuring  the  quality  of  software  program  management. 

In  this  research,  the  QMM  survey  instrument  is  used  to  measure  the  performance 
of  ten  program  managers  on  DoD  software  development  programs.  The  ten  program 
managers  are  asked  to  complete  the  survey  instrument.  In  addition,  the  survey  instrument 
is  given  to  one  to  two  development  team  members  on  each  program  who  are 
knowledgeable  about  the  overall  practices  and  success  of  the  program.  A  requirement  for 
choosing  the  individual  development  team  members  is  to  select  team  members  who  have 
a  good  understanding  of  the  overall  program  and  are  knowledgeable  about  the 
management  practices  and  infrastructure  implemented  by  the  program  manager 
throughout  the  program.  The  intent  is  to  choose  individuals  whose  experience  is  not 
limited  to  specific  areas  of  the  program.  The  individual  development  team  members  are 
asked  to  answer  the  QMM  survey  questions  as  they  pertain  to  the  program  management 
and  the  management  characteristics  of  the  program  manager  being  evaluated.  The 
program  manager  is  asked  to  determine  a  specific  point  in  time  on  the  program,  such  as  a 
milestone  or  delivery,  for  the  evaluation  of  the  program  management  and  to  define  it  such 
that  the  individual  development  team  members  will  be  able  to  identify  with  the  point  in 
time  that  was  selected  and  evaluate  the  program  for  that  same  point  in  time. 

The  survey  instrument  is  applied  to  the  interviewees  in  two  ways — either  via  a 

private  one-on-one  personal  interview  which  lasts  approximately  two  hours,  or  via  an 
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electronic  copy  which  is  distributed  by  email,  filled  out  by  the  interviewee  and  returned. 
In  eight  of  the  ten  programs  (17  surveys  total)  the  survey  instrument  is  applied  in 
personal  interviews,  in  the  remaining  two,  programs  H  and  J,  (four  surveys  total)  the 
survey  instrument  is  applied  through  email.  The  emailed  surveys  executed  later  in  the 
research  process  benefited  from  the  experience  gained  in  the  execution  of  the  surveys  via 
personal  interview.  By  the  time  three  of  the  four  emailed  surveys  were  distributed,  a 
short  data  dictionary  of  terms  on  the  survey  and  answers  to  frequently  asked  questions 
was  available  and  included  with  the  survey  distribution.  The  questionable  terms  in  the 
data  dictionary  and  frequently  asked  questions  were  identified  from  feedback  obtained 
during  the  in-person  interviews.  The  interviewees  surveyed  via  personal  interview, 
programs  A  -  G  and  program  I,  benefited  from  the  opportunity  to  ask  for  clarification  on 
terms  and  the  intent  of  individual  questions  during  the  interview  process. 

The  format  for  the  personal  interviews  included  scheduling  the  interview  in  a 
conference  room  or  a  quiet  place  away  from  interruptions,  executing  the  survey,  and 
asking  the  participant  to  subjectively  rate  the  success  of  the  program.  The  interviewees 
are  given  the  survey  instrument  on  paper  along  with  an  explanation  of  the  survey  format 
and  are  encouraged  to  request  clarification  at  any  time. 

Once  the  survey  is  completed,  the  interviewees  are  asked  to  subjectively  rate  the 

success  of  the  program,  at  the  point  in  time  the  program  is  being  evaluated,  on  a  scale  of 

zero  to  ten.  A  score  of  zero  corresponds  to  complete  and  utter  failure,  no  viable  product 

produced  and  the  program  failed  miserably.  A  score  of  ten  corresponds  to  complete 

program  success,  the  program  produced  an  outstanding  product,  on  time,  within  budget, 

with  no  bugs,  and  the  customer  was  ecstatic  with  the  quality  of  the  product.  To  assist  the 
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interviewees  in  visualizing  the  scale  the  researcher  draws  and  labels  a  scale  as  shown  in 
Figure  2  and  the  interviewees  are  asked  to  place  an  “X”  on  the  scale  which  represents  the 
success  of  the  program  at  the  evaluation  point  in  time.  Once  this  is  done,  the  researcher 
asks  the  interviewee  for  a  specific  numerical  value  to  associate  with  the  “X”  on  the  scale. 


Total  Perfect 

Failure  No  Errors 

Figure  2.  Overall  Program  Score  Scale. 

Finally  the  researcher  asks  the  interviewee  for  feedback  on  the  survey  instrument 
itself.  For  example,  where  there  any  questions  which  did  not  make  sense?  Are  there 
words  which  the  interviewee  did  not  understand?  Is  the  survey  instrument  too  long? 
How  would  they  improve  the  survey  instrument?  What  was  their  overall  impression  of 
the  survey  instrument?  This  feedback  is  collected  to  obtain  a  feel  for  the  level  of 
frustration  the  survey  induced  in  the  interviewee  and  to  provide  glimpses  into  the 
viability  of  the  survey  instrument  for  future  improvements. 

The  email  surveys  include  an  additional  survey  document  to  collect  background 
information  such  as  the  number  of  software  developers  working  on  the  program,  the  time 
frame  of  the  program  and  the  overall  program  success  score  of  the  interviewee.  The 
email  interviewees  are  encouraged  to  contact  the  researcher  via  telephone  or  with 
questions.  In  one  case,  the  email  survey  was  followed  up  by  a  telephone  call  to  gain 
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further  feedback  from  the  interviewee  regarding  the  survey  instrument.  In  this  case  the 
interviewee  had  very  strong  opinions  about  the  survey  instrument  and  wished  to  bring 
them  to  the  researcher’s  attention.  In  the  other  email  surveys  the  interviewees  respond  to 
the  additional  background  questions  through  email  alone. 

The  survey  instruments  utilized  in  both  the  personal  interview  surveys  and  the 
email  surveys  are  identical  in  content.  The  only  difference  in  the  survey  instruments  is 
the  medium  through  which  they  are  applied.  The  personal  interview  QMM  survey 
instruments  are  hard  copies  of  the  survey  instrument  which  is  emailed  with  the  exception 
of  the  additional  background-questions  document  and  the  data  dictionary  and  frequently- 
asked-questions  document.  The  emailed  survey  instruments  are  made  up  of  the  following 
files: 

•  Microsoft  Word  file  containing  Part  I  of  the  QMM  survey,  the  pair-choice 
questions. 

•  Microsoft  Excel  file  containing  Part  II  of  the  QMM  survey,  the  yes/no  questions. 

•  Microsoft  Word  file  containing  the  additional  background  questions. 

•  Microsoft  Word  file  containing  background  information  on  the  QMM  survey  and 
the  goals  of  the  current  research  for  the  interviewee’s  information. 

•  Microsoft  Word  file  containing  a  data  dictionary  of  QMM  terms  and  answers  to 
questions  frequently  asked  about  the  QMM  survey  instrument. 

Part  I  of  the  QMM  survey  instrument,  the  pair-choice  questions,  consists  of  two 
questions  side  by  side  on  a  single  line  with  a  column  beside  each  question.  The 
interviewee  is  asked  to  check  the  box  next  to  the  question  or  statement  on  each  line  which 
most  closely  reflects  what  is  or  was  happening  on  the  program  at  the  specific  point  in 
time  (milestone,  delivery,  etc.)  on  the  program  which  is  being  evaluated.  The 
interviewee  chooses  between  the  two  choices  on  each  line  of  the  survey.  The  statement 
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or  answer  which  most  closely  reflects  the  program  does  not  need  to  be  an  exact  match. 
Each  pair  statement  represents  two  differing  ideas  in  an  effort  to  ascertain  a  tendency  of 
the  individual  in  the  area  of  interest,  such  as  the  use  of  formal  requirements 
documentation  versus  informal  or  no  documentation.  Often  the  pair  choices  are  repeated 
with  different  wording  to  confirm  earlier  choices  and  measure  the  strength  of  the 
tendency  .  The  survey  format,  with  the  proper  mix  of  questions  and  variation  repetitions 
is  intended  to  be  used  to  reach  consensus  on  issues  and  measure  the  strength  of 
tendencies  [Ref.  7], 

Part  II  of  the  QMM  survey  instrument,  the  yes/no  questions,  consists  of  a  single 
question  per  line  with  three  columns  next  to  it  labeled  “yes,”  “no,”  and  “n/a” 
respectively.  The  interviewee  answers  the  question  as  it  pertains  to  the  program  manager 
and  the  program  at  the  specific  point  in  time  the  program  is  being  evaluated  by  placing  a 
check  in  the  appropriate  “yes”,  “no”,  or  “n/a”  box  next  to  the  question.  The  use  of  the 
“n/a”  box  is  discouraged.  However,  it  exists  and  is  used  in  the  cases  in  which  the 
program  manager  does  not  have  direct  control  over  the  issue  which  is  asked  in  the 
question.  For  example,  when  an  interviewee  evaluates  a  government  program  manager 
who  is  in  charge  of  a  contractor  developing  the  software,  the  government  program 
manager  does  not  have  direct  hire/fire  authority  over  the  development  personnel.  In  the 
case  in  which  the  survey  question  asks  if  the  program  manager  has  direct  hire/fire 
authority  over  the  personnel,  the  appropriate  answer  would  be  “n/a”.  The  “n/a ”  answer 
column  is  present  to  allow  the  interviewee  to  indicate  the  program  manager  is  constrained 
in  the  area  being  surveyed  and  does  not  penalize  the  program  manager  for  factors  beyond 
their  control. 
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The  weighting  of  the  questions  on  the  survey  instrument  are  the  same  as  used  by 
Martin  Machniak  [Ref.  7].  The  pair-choice  questions  in  the  requirements  management 
section  have  a  possible  score  of  zero  to  two  with  different  upper  bounds  on  the  score  of 
each  question  based  upon  its  relative  weight  and  importance  in  the  section.  The  possible 
scores  in  the  estimation/planning  management,  people  management,  and  risk 
management  sections  have  a  possible  score  of  zero  to  one.  The  yes/no  questions  have  a 
possible  score  of  minus  four  to  four  with  different  upper  and  lower  bounds  on  the  score 
of  each  question  based  upon  its  relative  weight  and  importance  in  the  section. 

To  score  the  results  of  Part  I  and  Part  II  of  the  QMM  survey  instrument,  the 
interviewee  responses  are  entered  into  a  Microsoft  Excel  spread  sheet  containing  the 
QMM  questions,  response  columns,  and  question  weighting  factors.  A  separate  spread 
sheet  file  is  created  for  each  interviewee’s  results  for  each  part  of  the  QMM  survey. 
Thus,  interviewee  A  -  PM  Will  have  an  Excel  file  containing  the  results  for  their  QMM 
Part  I  results  and  a  separate  file  containing  their  QMM  Part  II  results.  The  spread  sheets 
automatically  calculate  the  total  points  for  each  of  the  four  QMM  sections:  requirements 
management,  estimation/planning  management,  people  management,  and  risk 
management.  These  results,  along  with  the  interviewee’s  subjective  overall  program 
score,  are  entered  manually  in  a  QMM  summary  spread  sheet  for  each  interviewee  which 
then  calculates  the  total  points  for  each  QMM  section,  the  overall  QMM  score,  and  the 
QMM  percentage  score. 

The  summary  data  for  each  program  and  each  interviewee  is  then  manually 

entered  into  an  Excel  summary  spread  sheet.  The  data  contained  in  the  summary  spread 

sheet  identifies  the  program,  the  interviewee  for  the  program,  and  the  interviewee’s 
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scores  for  the  following  QMM  sections:  requirements  management,  estimation/planning 
management,  people  management,  and  risk  management,  QMM  score,  QMM  percentage 
score,  and  overall  program  score.  The  data  in  the  summary  data  spread  sheets  is  then 
used  for  analyzing  the  data. 

B.  ANALYSIS  TECHNIQUES 

The  data  analysis  consists  of  graphing  the  data  to  look  for  obvious  trends, 
computing  the  mean,  standard  deviation,  minimum  and  maximum  values  for  the  data 
range,  variance,  and  various  data  set  correlation  to  explore  the  data  further. 

The  QMM  point  totals  for  the  four  sections — requirements  management, 
estimation/planning  management,  people  management,  and  risk  management — are 
converted  into  percentages  and  constrained  to  a  scale  of  zero  to  ten  by  the  following 
steps.  The  total  points  scored  for  each  section  are  normalized  to  zero  by  subtracting  the 
minimum  possible  points  for  each  section  from  the  points  scored  for  the  section.  The 
resulting  normalized  score  is  then  divided  by  total  of  the  maximum  point  totals  possible 
in  each  section  minus  the  minimum  possible  points  for  each  section  to  compute  the 
percentage  of  the  points  scored  in  each  section.  These  section  score  percentages  are  then 
multiplied  by  a  factor  of  ten  to  convert  the  percentages  from  a  scale  of  zero  to  100 
percent  to  a  scale  of  zero  to  ten.  This  is  done  to  keep  the  scales  of  all  the  bar  charts  the 
same,  providing  for  easier  direct  comparisons  with  the  overall  program  scores  which  are 
already  constrained  to  the  scale  of  zero  to  ten.  The  equations  are  as  follows: 
SECTIONmax  -  SECTIONmin  =  SECTIONtpp 

SECTIONscore  -  SECTIONmin  =  SECTIONSCore  normalized 
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(SECTIONscore  normalized/SECTIONtpp)  -  SECTIONpercentage  score 
SECTIONpercentage  score  *  10-0  =  SECTIONscore  scaled  from  o  to  10 


where 

SECTIONmax  is  the  maximum  number  of  points  possible  for  the  section 
SECTIONmin  is  the  minimum  number  of  points  possible  for  the  section 
SECTIONjpp  is  the  total  points  possible  for  the  section 

SECTIONscore  normalized  is  the  total  points  scored  for  the  section  normalized  to 
zero 

SECTIONpercentage  score  is  the  percentage  of  the  points  scored  for  the  section 
SECTIONscore  scaled  from  o  to  10  is  the  percentage  score  for  the  section 
converted  to  a  scale  of  zero  to  ten 

For  graphing  purposes  and  making  direct  comparisons  with  the  overall  program 
score,  the  QMM  percentage  score  is  divided  by  a  factor  of  ten  to  constrain  it  to  a  scale  of 
zero  to  ten.  Thus,  the  summary  data  tables  and  bar  charts,  in  Chapter  IV  and  Appendix  E 
respectively,  report  the  QMM  percentage  scores  on  a  scale  of  zero  to  ten  with  a  score  of 
zero  corresponding  to  a  QMM  percentage  score  of  zero  percent  and  a  score  of  ten 
corresponding  to  a  QMM  percentage  score  of  100  percent. 

The  graphs  created  are  bar  charts  displaying  the  values  of  all  the  surveyed 
programs  for  the  following  data  sets: 

•  Program  manager’s  (PM)  overall  program  score  and  PM  QMM  percentage  score 

•  Individual  development  team  member’s  (IND)  overall  program  score  and  IND 
QMM  percentage  score 

•  PM  overall  program  success  score,  PM  QMM  percentage  score,  IND  overall 
program  success  score,  and  IND  QMM  percentage  score 
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•  PM  and  IND  overall  program  success  scores 

•  PM  and  IND  QMM  percentage  scores 

•  PM  and  IND  requirements  management  scores 

•  PM  and  IND  estimation/planning  scores 

•  PM  and  IND  people  management  scores 

•  PM  and  IND  risk  management  scores 


The  data  analysis  includes  computation  of  the  mean,  minimum  value,  maximum 
value,  standard  deviation,  and  variance  for  the  following  data  sets: 

•  PM  overall  program  scores  of  all  the  programs 

•  PM  QMM  percentage  scores  of  all  the  programs 

•  PM  requirements  management  scores  of  all  the  programs 

•  PM  estimation/planning  management  scores  of  all  the  programs 

•  PM  people  management  scores  of  all  the  programs 

•  PM  risk  management  scores  of  all  the  programs 

•  IND  overall  program  scores  of  all  the  programs 

•  IND  QMM  percentage  scores  of  all  the  programs 

•  IND  requirements  management  scores  of  all  the  programs 

•  IND  estimation/planning  management  scores  of  all  the  programs 

•  IND  people  management  scores  of  all  the  programs 

•  IND  risk  management  scores  of  all  the  programs 


The  data  analysis  also  computes  the  correlation  between  the  following  data  sets: 

•  PM  overall  program  scores  and  PM  QMM  percentage  scores 

•  PM  overall  program  scores  and  PM  requirements  management  scores 

•  PM  overall  program  scores  and  PM  estimation/planning  management 

•  PM  overall  program  scores  and  PM  people  management  scores 

•  PM  overall  program  scores  and  PM  risk  management  scores 

•  PM  QMM  percentage  scores  and  PM  requirements  management  scores 

•  PM  QMM  percentage  scores  and  PM  estimation/planning  management 

•  PM  QMM  percentage  scores  and  PM  people  management  scores 

•  PM  QMM  percentage  scores  and  PM  risk  management  scores 

•  PM  requirements  management  and  PM  estimation/planning  management 

•  PM  requirements  management  and  PM  people  management  scores 

•  PM  requirements  management  and  PM  risk  management  scores 

•  PM  estimation/planning  management  and  PM  people  management  scores 

•  PM  estimation/planning  management  and  PM  risk  management  scores 
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PM  people  management  and  PM  risk  management  scores 


•  IND  overall  program  scores  and  IND  QMM  percentage  scores 

•  IND  overall  program  scores  and  IND  requirements  management  scores 

•  IND  overall  program  scores  and  IND  estimation/planning  management 

•  IND  overall  program  scores  and  IND  people  management  scores 

•  IND  overall  program  scores  and  IND  risk  management  scores 

•  IND  QMM  percentage  scores  and  IND  requirements  management  scores 

•  IND  QMM  percentage  scores  and  IND  estimation/planning  management 

•  IND  QMM  percentage  scores  and  IND  people  management  scores 

•  IND  QMM  percentage  scores  and  IND  risk  management  scores 

•  IND  requirements  management  and  IND  estimation/planning  management 

•  IND  requirements  management  and  IND  people  management  scores 

•  IND  requirements  management  and  IND  risk  management  scores 

•  IND  estimation/planning  management  and  IND  people  management  scores 

•  IND  estimation/planning  management  and  IND  risk  management  scores 

•  IND  people  management  and  IND  risk  management  scores 

•  PM  overall  program  scores  and  IND  overall  program  scores 

•  PM  QMM  percentage  scores  and  IND  QMM  percentage  scores 

•  PM  requirements  management  scores  and  IND  requirements  management  scores 

•  PM  estimation/planning  management  scores  and  IND  estimation/planning 
management  scores 

•  PM  people  management  scores  and  IND  people  management  scores 

•  PM  risk  management  scores  and  IND  risk  management  scores 
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IV.  RESULTS  AND  ANALYSIS 


A.  RESEARCH  BACKGROUND 

In  order  to  encourage  complete  and  open  participation  in  the  QMM  survey,  the 
interviewees  are  assured  at  the  beginning  of  the  survey  process  that  the  results  of  the 
surveys  will  be  reported  anonymously.  To  this  end,  the  program  data  reported  here  is 
reported  as  program  A,  B,  C,  etc.,  and  the  letters  associated  with  the  individual  programs 
do  not  reflect  the  order  in  which  the  programs  were  surveyed.  Appendix  B  contains  the 
responses  to  the  survey  for  all  of  the  participants. 

The  ten  programs  surveyed  range  in  size  from  a  minimum  of  three  software 
developers  to  a  maximum  of  twenty-five  software  developers;  see  Figure  3.  The  time 
frame  of  the  programs  surveyed  range  from  1992  to  present  day,  with  most  of  them 
within  the  last  two  years;  see  Figure  4.  All  of  the  programs  are  DoD  software 
development  efforts. 

The  minimum  amount  of  time  required  to  complete  the  survey  instrument  in  a 
personal  interview  is  an  hour.  The  average  time  required  to  complete  the  survey 
instrument  is  two  hours,  and  the  longest  time  required  is  four  hours. 
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B.  GRAPH  ANALYSIS 


To  allow  comparison  between  the  different  data  types,  a  standard  scale  of  zero  to 
ten  was  chosen  with  the  minimum,  zero,  corresponding  to  complete  and  utter  program 
failure  and  zero  percent  of  the  total  points  possible  for  the  QMM  percentage  score  and 
the  QMM  section  scores.  The  maximum  possible  score  is  ten  which  corresponds  to 
complete  program  success  and  100  percent  of  the  total  points  possible  for  the  QMM 
percentage  score  and  the  QMM  sections. 

The  QMM  survey  data  is  shown  as  bar  charts  in  Appendix  C,  Figures  Cl  -  C9. 

The  bar  charts  are  examined  to  determine  if  there  are  any  obvious  trends.  The 
possibility  of  the  program  managers  consistently  rating  the  success  of  the  program  higher 
than  the  corresponding  QMM  percentage  score  is  not  found  in  the  graphs,  although  it  was 
expected.  One  instance  of  the  program  manager  rating  the  program  overall  far  higher 
than  the  QMM  percentage  score  and  the  individual  development  team  member  overall 
program  score  and  QMM  percentage  score  is  shown  in  program  A;  see  Figure  C3.  A 
possible  explanation  for  this  is  self-enhancement  bias  on  the  part  of  the  program 
manager. 

An  exciting  result  of  the  research  is  seen  in  the  data  for  program  F;  see  Figure  C3. 

Program  F  does  not  utilize  many  of  the  currently  accepted  software  engineering 

management  processes  and  procedures  such  as  formal  risk  management.  The  program  is 

ranked  as  highly  successful  by  the  software  developers  and  the  customers  with  an  overall 

program  success  score  of  9.0,  yet  the  QMM  percentage  score  consistently  ranks  the 

program  as  a  5.4.  The  exciting  part  of  the  discovery  is  that  it  appears  the  QMM  survey 
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instrument  does  measure  the  implementation  and  the  successfulness  of  the 
implementation  of  currently  accepted  software  engineering  management  practices  as 
defined  by  Martin  Machniak  [Ref.  7]. 

In  programs  C,  I,  and  J,  the  program  manager’s  QMM  percentage  score  is 
consistently  higher  than  the  program  manager’s  overall  program  success  score  and  the 
scores  of  the  individual  development  team  member  in  the  areas  of  overall  program 
success  score  and  QMM  percentage  score;  see  Figure  C3.  A  possible  explanation  for  this 
discrepancy  is  that  the  program  manager  believes  he  or  she  is  implementing  and  using 
more  of  the  software  engineering  management  practices  than  are  actually  being 
implemented  at  the  development-team  level.  If  this  is  the  case,  then  the  QMM  survey 
scores  could  be  used  to  assist  the  program  manager  in  identifying  differences  in 
perception  between  themselves  and  the  rest  of  the  development  team,  and  take  steps  to 
improve  the  communication  within  the  development  team.  Better  communication  could 
assist  the  program  manager  to  better  guide  the  program  development  team’s  efforts  and 
receive  feedback  on  the  processes  and  procedures  which  the  program  manager 
implements  to  enable  the  team  to  improve  them. 


C.  DATA  ANALYSIS 

A  summary  of  the  program  manager  and  independent  development  team  member 
data  for  all  the  programs  surveyed  is  presented  in  Table  3  and  Table  4.  In  the  tables,  the 
QMM  percentage  score,  requirements  management,  estimation/planning  management, 
people  management,  and  risk  management  scores  are  all  normalized  to  a  scale  of  zero  to 
ten.  A  score  of  zero  corresponds  to  zero  percent  or  zero  percent  of  the  points  possible  for 
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the  section.  A  score  of  ten  corresponds  to  100  percent  or  100  percent  of  the  points 
possible  for  the  section. 

The  correlations  of  the  various  data  sets  presented  in  the  following  tables  are 
computed  using  the  coefficient  of  correlation,  p,  where  -1  <  p  <  1.  A  value  of  p  =  -1 
implies  a  perfect  straight  line  relationship  between  the  correlated  data  sets  with  a  negative 
slope.  A  value  of  p  =  1  implies  a  perfect  straight  line  relationship  between  the  correlated 
data  sets  with  a  positive  slope.  A  value  of  p  =  0  implies  no  linear  relationship  between 
the  correlated  data  sets. 

The  data  analysis  for  the  program  manager  data  is  presented  in  Table  5.  From  this 
data  it  can  be  seen  that  there  exists  a  relatively  high  divergence  of  2.09  in  the  overall 
program  scores  for  the  various  programs.  This  is  expected  as  the  programs  all  have 
different  management,  requirements,  and  work  forces,  and  are  constrained  by  different 
budgets  and  schedules. 


Program 

PM 

Program 

Score 

PM 

QMM 

Score 

PM  Req. 
Man. 

PM  Est.  / 
Planning 

PM 

People 

Man. 

PM  Risk 
Man. 

A 

9.9 

6.8 

5.2 

6.2 

8.6 

4.0 

B 

7.5 

8.9 

8.6 

9.3 

8.8 

9.5 

C 

8 

8.1 

8.8 

7.5 

8.8 

5.1 

D 

7 

7.0 

5.0 

7.9 

5.9 

E 

4 

5.2 

5.2 

4.5 

6.1 

2.6 

F 

9 

5.5 

3.7 

■ZS 

0.7 

G 

7 

6.8 

6.0 

6.6 

7.8 

4.9 

H 

6 

7.7 

7.1 

7.5 

7.8 

8.3 

1 

3 

4.8 

4.8 

3.8 

6.3 

0.9 

J 

7 

8.2 

8.6 

6.9 

8.4 

8.1 

Table  3.  Program  Manager  Summary  Data. 
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Program 

IND 

Program 

Score 

IND 

QMM 

Score 

IND 

Req. 

Man. 

IND  Est.  / 
Planning 

IND 

People 

Man. 

IND 

Risk 

Man. 

A 

7 

5.7 

6.2 

5.1 

5.6 

5.1 

B 

8 

8.7 

8.6 

9.0 

8.5 

9.0 

C 

5 

6.4 

8.2 

4.0 

7.8 

1.7 

D 

7 

5.2 

8.1 

3.8 

4.1 

5.6 

D 

7.5 

7.7 

8.0 

9.3 

6.5 

8.9 

E 

6 

6.5 

6.6 

6.8 

6.6 

5.2 

F 

9 

5.4 

7.0 

3.5 

5.8 

3.3 

G 

8 

7.0 

6.7 

6.7 

8.1 

4.3 

H 

6.5 

7.7 

6.9 

7.5 

7.9 

8.4 

1 

3 

2.6 

3.2 

2.2 

2.5 

1.1 

J 

7 

6.3 

7.7 

7.2 

5.7 

4.3 

Table  4.  Individual  Development  Team  Member  Summary  Data. 


PM 

Program 

Score 

PM 

QMM 

Score 

PM  Req. 
Man. 

PM  Est.  / 
Planning 

PM 

People 

Man. 

PM  Risk 
Man. 

Mean 

6.84 

6.91 

6.70 

6.11 

4.99 

mm 

9.90 

8.93 

8.77 

9.34 

8.81 

9.46 

Low 

3.00 

4.84 

4.77 

3.65 

6.06 

0.72 

Standard 

Deviation 

2.09 

1.38 

1.57 

1.85 

0.98 

3.04 

Variance 

4.38 

1.90 

2.48 

3.41 

0.96 

9.23 

Table  5.  Program  Manager  Data  Analysis. 


The  standard  deviation  of  the  program  managers  in  the  area  of  people 
management  is  a  relatively  low  value  of  0.98  with  a  corresponding  mean  of  7.77.  This 
indicates  that  the  program  managers  believe  they  are  good  people  managers  or  at  least 
believe  they  are  implementing  many  of  the  people  management  policies  and  procedures 
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the  QMM  survey  instrument  measures.  This  is  further  supported  by  the  strong 
correlation  between  the  program  manager  overall  program  score  and  the  people 
management  score  of  0.771  in  Table  6. 

The  program  manager  estimation/planning  management  and  risk  management 
sections  have  high  standard  deviations  of  1.85  and  3.04,  shown  in  Table  5,  along  with  a 
strong  correlation  of  0.886  for  these  two  sections,  shown  in  Table  6.  These  management 
areas  involve  a  great  deal  of  uncertainty.  The  data  indicates  that  program  managers  who 
do  well  managing  the  uncertainty  in  one  area  generally  do  well  managing  the  uncertainty 
in  the  other  area  as  well. 


PM 

QMM 

Score 

PM  Req. 
Man. 

PM  Est.  / 
Planning 

PM 

People 

Man. 

PM  Risk 
Man. 

PM 

Program 

0.44694 

0.27539 

0.31676 

0.77114 

0.19311 

PM  QMM 
Score 

0.89919 

0.93389 

0.88371 

0.92171 

PM  Req. 
Man. 

0.7612 

0.73 

0.80035 

0.77171 

0.88561 

PM 

People 

0.67601 

Table  6.  Program  Manager  Data  Correlation. 


In  Table  6,  the  areas  of  requirements  management  and  risk  management  show  the 
weakest  correlation  with  the  overall  program  score  with  correlations  of  0.275  and  0.193. 
This  indicates  the  implementation  of  the  requirements  management  and  risk  management 
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practices  do  not  play  a  big  role  in  the  subjective  evaluation  of  the  program  manager  on 
the  success  of  the  overall  program. 

The  program  manager  data  correlations  of  the  four  QMM  sections — requirements 
management,  estimation/planning  management,  people  management,  and  risk 
management — are  strongly  correlated  with  the  program  manager  QMM  percentage  score. 
This  is  expected  since  the  QMM  percentage  score  is  comprised  of  the  four  section  scores. 
However,  the  strengths  of  the  correlations  are  surprising.  The  strongest  correlation  is 
expected  to  be  the  correlation  with  the  people  management  section  as  this  section  has  the 
highest  Importance  Coefficient  (IC)  of  1.86.  Yet,  the  people  management  section  has  the 
weakest  correleation  of  the  four  sections  with  a  correlation  of  0.884  and  the  sections  with 
the  lowest  IC  values — estimation/planning  management  and  risk  management — have  the 
strongest  correlations  of  0.933  and  0.922;  see  Table  6.  These  results  indicate  that  the 
QMM  percentage  scores  for  the  program  managers  are  driven  more  by  the  scores  in  the 
estimation/planning  management  and  risk  management  sections.  This  is  not  surprising 
given  the  fact  that  the  program-manager  standard  deviation  for  the  people-management 
section  is  low  with  a  value  of  0.98,  and  the  mean  for  the  section  is  relatively  high  with  a 
value  of  7.77.  The  people  management  scores  for  the  various  program  managers  are 
relatively  constant.  Therefore  the  factors  which  vary  the  most  and  distinguish  the  QMM 
percentage  scores  for  the  various  program  managers  are  the  scores  for  the  areas  with  the 
highest  standard  deviation,  that  is,  the  estimation/planning  management  and  risk 
management  sections. 
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In  general,  the  correlation  analysis  for  the  program  manager  data  indicates  that, 
except  for  the  subjective  overall  program  success  score,  all  the  variables  have  a  strong 
correlation. 

Table  7  presents  the  results  of  the  analysis  of  the  data  gathered  from  surveying  the 
individual  development  team  members.  The  standard  deviation  for  the  people 
management  section  is  higher,  1.84,  than  the  program  manager  standard  deviation  for  the 
same  section,  0.98.  The  mean  for  the  individual  development  team  member  data  is  6.27 
versus  7.77  for  the  program  manager  data.  This  indicates  the  individual  development 
team  members  either  do  not  think  the  program  managers  are  as  good  at  managing  people 
as  the  program  managers  think  they  are  or  the  individual  development  team  members  do 
not  see  or  do  not  understand  the  people-management  procedures  and  practices  the 
program  managers  believe  they  have  implemented  on  the  program. 


IND 

Program 

Score 

IND 

QMM 

Score 

IND 

Req. 

Man. 

IND  Est.  / 
Planning 

IND 

People 

Man. 

IND 

Risk 

Man. 

Mean 

6.73 

6.29 

7.02 

5.92 

6.27 

5.18 

■iil-Ul 

9.00 

8.71 

8.65 

9.34 

8.50 

9.04 

Low 

3.00 

2.57 

3.16 

2.22 

2.50 

1.14 

Standard 

Deviation 

1.63 

1.64 

1.49 

2.35 

1.84 

2.70 

Variance 

2.67 

2.68 

2.23 

5.51 

3.37 

7.30 

Table  7.  Individual  Development  Team  Member  Data  Analysis. 


The  high  standard  deviations  for  estimation/planning  management  and  risk 
management  in  Table  7  mirror  the  results  for  the  program  manager  data  analysis. 
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Perhaps  this  is  due  to  these  sections  having  more  uncertainty  involved  in  the  management 
of  them. 

The  individual  development  team  member  data  correlations;  see  Table  8,  between 
the  four  QMM  sections  and  the  QMM  percentage  score,  are  all  strong.  However,  the 
strongest  correlation  between  the  people  management  section  and  the  QMM  percentage 
score  of  0.902,  is  expected  based  upon  the  IC  coefficients  for  the  four  sections.  What  is 
unexpected  is  the  requirements  management  section  has  a  weaker  correlation  than  either 
the  estimation/planning  management  or  risk  management  section. 

The  strong  correlation  between  the  estimation/planning  management  and  risk 
management  sections  of  0.840  in  Table  8  indicates  that  the  program  managers  who  are 
good  at  managing  the  uncertainty  in  one  area  are  also  good  at  managing  the  uncertainty 
in  the  other  area. 
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0.46586 
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IND  QMM 
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0.78029 

IND  Req. 
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0.53456 

0.62857 

0.5262 

IND  Est.  / 
Planning 

0.66453 

0.84014 

IND 

People 

0.48039 

Table  8.  Individual  Development  Team  Member  Data  Correlation. 
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Table  9  presents  the  results  of  data  correlations  between  the  program  manager 
data  and  the  individual  development  team  member  data.  The  correlation  of  the  QMM 
percentage  scores  for  the  program  manager  and  the  individual  development  team 
members  is  a  strong  0.693. 

The  correlation  between  the  requirements  management  and  risk  management 
sections  in  Table  9  are  strong:  0.769  and  0.686,  respectively.  This  indicates  that  perhaps 
these  two  areas  are  the  easiest  to  gain  insight  into  for  non-program  managers  on  the 
practices  and  procedures  implemented  by  the  program  manager.  The  lowest  correlation 
is  the  people-management  section. 
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0.69338 
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0.76933 
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0.47345 

IND  Risk 
Man. 
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Table  9.  Program  Manager  and  Individual  Development  Team  Member  Data 

Correlation. 
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The  data  analysis  and  data  correlations  are  performed  on  the  research  data  without 
the  inclusion  of  program  F  as  this  program  does  not  use  many  of  the  recommended 
software  engineering  management  practices  such  as  formal  risk  assessment  and 
management.  The  results  of  the  analysis  are  presented  in  Tables  10  -  14. 

The  exclusion  of  program  F  from  the  data  set  shows  an  increase  in  the  strength  of 
the  correlation  between  the  program  manager’s  overall  program  score  and  the  QMM 
percentage  score  from  0.447  to  0.673  as  shown  in  Table  11.  The  data  for  the  program 
managers  continues  to  show  a  very  strong  correlation  between  the  overall  program  score 
and  the  people-management  section.  The  high  standard  deviations  for  the 
estimation/planning  management  and  risk  management  sections  along  with  the  strong 
correlation  between  these  two  sections  also  continues  to  be  present;  see  Table  10  and 
Table  11.  The  program  manager  data  continues  to  show  strong  correlations  between  all 
the  variables  as  was  seen  in  the  previous  data  set;  see  Table  11. 
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6.82 
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7.82 

5.46 

9.90 

8.93 

8.77 

9.34 

8.81 

9.46 
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3.00 

4.84 

4.77 

3.77 

6.06 

0.90 

Standard 

Deviation 

2.07 

1.36 

1.61 

1.73 

1.03 

2.80 

Variance 

4.28 

1.84 

2.61 

3.00 

1.06 

7.85 

Table  10.  Program  Manager  Data  Analysis  Without  Program  F. 
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PM 

QMM 

Score 
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PM  Req. 
Man. 

0.7506 

0.72161 

0.80154 

M 

0.79726 

0.85182 

PM 

People 

Man. 

0.69386 

Table  1 1 .  Program  Manager  Data  Correlation  Without  Program  F. 


The  data  for  the  individual  development  team  members  shows  the  same  trends  as 
in  the  previous  data  with  the  following  exceptions.  The  correlation  between  the  overall 
program  score  and  the  QMM  percentage  score  has  increased  from  0.589  to  a  strong 
0.772;  see  Table  13. 
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8.50 
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3.16 

2.22 

2.50 
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1.53 

1.70 

1.57 

2.33 

1.93 

2.77 
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2.33 

2.88 

2.48 

5.42 

3.71 

7.66 

Table  12.  Individual  Development  Team  Member  Data  Analysis  Without  Program 

F. 
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Table  13.  Individual  Development  Team  Member  Data  Correlation  Without 

Program  F. 
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People 
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Table  14.  Program  Manager  and  Individual  Development  Team  Member  Data 

Correlation  Without  Program  F. 
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Additionally,  the  correlation  between  the  overall  program  score  and  the  areas  of 
requirements  management,  estimation/planning  management,  and  risk  management  have 
increased  significantly  to  0.708, 0.745,  and  0.714  respectively;  see  Table  13. 

Since  the  program  manager’s  overall  program  score  for  program  A  appears  to  be 
unusually  high,  9.9,  possibly  due  to  enhancement  bias,  the  data  analysis  and  data 
correlations  have  been  computed  without  the  inclusion  of  the  program  A  and  program  F 
data.  These  results  are  presented  in  Tables  15  -  19. 

The  program  manager  data  continues  to  show  the  same  trends  as  in  the  previous 
data  analysis  with  the  following  exceptions.  The  overall  program  score  now  correlates 
very  strongly  with  the  four  QMM  sections;  see  Table  16.  The  overall  program  score 
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PM  Req. 
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Mean 
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4.84 
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0.90 
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1.45 

1.59 
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1.05 
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3.14 

2.10 

2.53 

3.42 

1.10 

8.62 

Table  15.  Program  Manager  Data  Analysis  Without  Program  A  and  F. 


correlation  with  the  QMM  percentage  score  for  the  program  manager  has  increased  from 
0.447  in  the  original  data  set  to  0.673  in  the  data  set  without  program  F  to  the  present 
value  of  0.891;  see  Table  16.  This  indicates  the  QMM  survey  instrument  produces  very 
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good  QMM  percentage  score  results.  Additionally,  all  variables  for  the  program  manager 
correlate  very  strongly  with  each  other. 
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Table  16.  Program  Manager  Data  Correlation  Without  Program  A  and  F. 
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9.04 

Low 

3.00 

2.57 

3.16 

2.22 

2.50 

1.14 

Standard 

Deviation 

1.71 

1.90 

1.74 

2.58 

2.14 

3.10 
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Table  17.  Individual  Development  Team  Member  Data  Analysis  Without  Program  A 

andF. 
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Table  18.  Individual  Development  Team  Member  Data  Correlation  Without 

Program  A  and  F. 
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Table  19  Program  Manager  and  Individual  Development  Team  Member  Data 
Correlation  Without  Program  A  and  F. 
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The  individual  development  team  member  data  continues  to  show  the  same  trends 
as  in  the  previous  data  analysis  with  the  following  exceptions.  The  overall  program  score 
correlation  with  the  QMM  percentage  score  and  the  four  QMM  sections  has  increased  in 
strength;  see  Table  18.  The  correlation  of  the  overall  program  score  with  the  QMM 
percentage  score  has  increased  from  0.589  in  the  original  data  set  to  0.772  in  the  data  set 
without  program  F  data,  to  the  present  strong  correlation  value  of  0.817;  see  Table  18. 
This  indicates  the  QMM  survey  instrument  is  producing  very  good  QMM  percentage 
score  results. 

Although  the  data  correlations  within  the  program  manager  data  set  and  the 
individual  development  team  member  data  sets  are  very  strong,  the  correlation  between 
the  two  data  sets  as  shown  in  Table  19  remains  weaker.  The  strongest  correlation  is 
between  the  program  manager  and  individual  development  team  member  data  for 
requirements  management.  Requirements  management  is  an  area  which  by  necessity 
must  be  managed  on  the  program  to  understand  and  clarify  the  scope  and  goals  of  the 
program.  This  area  is  then  perhaps  more  visible  to  the  entire  development  team  whereas 
the  other  management  areas  do  not  have  the  same  level  of  visibility  and  insight 
throughout  the  development  team. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FUTURE 

WORK 


A.  CONCLUSIONS 

The  QMM  survey  instrument  yields  a  strong  correlation  with  the  QMM 
percentage  score  and  the  overall  program  score  for  both  the  program  manager  and 
individual  development  team  member  data  sets.  This  indicates  the  QMM  survey 
instrument  is  a  viable  instrument  for  measuring  the  quality  of  management  on  a  software- 
development  program  where  the  management  policies  and  procedures  are  the  same  as  or 
similar  to  the  currently  accepted  software  engineering  management  practices,  as  defined 
by  Martin  Machniak  [Ref.  7].  In  the  case  where  the  management  practices  and 
procedures  are  known  to  diverge  from  the  currently  accepted  practices  the  QMM  survey 
instrument  may  be  used  to  measure  the  level  to  which  the  currently  accepted  practices  are 
implemented  but  will  yield  a  lower  QMM  percentage  score  and  probably  predict  a  lower 
success  rate  for  the  program  than  the  actual  rate. 

The  QMM  survey  instrument  may  be  useful  in  detecting  discrepancies  between 
the  program  manager’s  and  the  individual  team  member’s  perspectives  and 
understanding.  When  the  program  manager’s  QMM  percentage  score  is  consistently 
higher  than  their  overall  program  score,  the  overall  program  success  score  and  QMM 
percentage  of  the  individual  development  team  member,  it  may  indicate  the  program 
manager  believes  he  or  she  is  implementing  and  successfully  using  more  software 
engineering  management  practices  on  the  program  than  are  actually  being  implemented 
on  the  development  team  level. 
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The  low  standard  deviation  and  high  mean  of  the  program  managers  in  the  area  of 
people  management  indicates  that  the  program  managers  believe  they  are  good  people 
managers.  Further  evidence  of  this  is  seen  in  the  strong  correlation  of  the  program 
manager’s  overall  program  score  and  the  people  management  score. 

A  higher  standard  deviation  and  lower  mean  in  the  area  of  people  management 
from  the  individual  development  team  member  data  than  the  program  manager  data 
indicates  that  the  individual  development  team  members  do  not  believe  the  program 
managers  are  as  good  at  managing  people  as  the  program  managers  themselves. 

The  estimation/planning  management  and  risk  management  sections  consistently 
have  high  standard  deviations  in  both  the  program  manager  and  individual  development 
team  member  data  and  strong  correlation  between  these  two  sections  as  well.  This 
indicates  that  there  exists  a  great  deal  of  uncertainty  in  managing  these  areas  and  that 
program  managers  who  are  good  at  managing  the  uncertainty  in  one  of  these  areas  are 
usually  good  at  managing  the  uncertainty  in  the  other  area  as  well. 

The  correlations  between  QMM  percentage  score  and  the  four  QMM  section 
scores  is  strong  for  both  the  program  manager  data  and  the  individual  development  team 
data  in  all  three  data  analysis  phases.  However,  the  correlation  between  the  program 
manager  and  the  individual  development  team  member  data  is  much  weaker  except  in  the 
area  of  requirements  management.  This  indicates  that  perhaps  the  area  of  requirements 
management  is  more  visible  and  better  understood  by  the  entire  team  than  other  areas  of 
management. 
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The  QMM  percentage  scores  have  a  strong  correlation  with  the  overall  program 
scores  for  both  the  program  manager  and  the  individual  development  team  member  data. 
This  is  particularly  true  for  the  data  set  which  excludes  the  program  A  and  F  data.  In  this 
data  set  the  correlation  for  the  program  manager  is  0.891  and  the  correlation  for  the 
individual  development  team  member  is  0.817.  This  strong  correlation  indicates  the 
QMM  survey  instrument  is  producing  very  good  QMM  percentage  score  results. 

B.  RECOMMENDATIONS  FOR  FUTURE  WORK 

The  recommendations  for  future  work  include  updating  and  further  testing  of  the 

QMM  survey  instrument,  developing  a  mechanism  to  provide  the  program  managers  with 
feedback  on  their  management  performance  and  possible  courses  of  action  to  take  to 
improve  their  performance,  and  integrating  the  QMM  survey  instrument  results  into 
existing  cost,  schedule  and  risk  models  to  improve  program  estimation  accuracy. 

Updating  the  QMM  survey  instrument  questions  includes  updating  the  focus  of 
the  survey  instrument,  refining  the  question  wording,  and  refining  the  question  weighting. 
As  advances  and  improvements  are  made  in  the  area  of  software  engineering 
management,  the  QMM  survey  instrument  will  need  to  be  updated  to  reflect  these 
advances  and  improvements.  The  wording  of  the  questions  should  be  further  refined  so 
the  questions  do  not  lead  the  interviewee  in  making  their  choice  between  two  answers 
and  so  the  clarity  of  the  question  intent  is  improved.  Adjustment  of  the  weighting  of  the 
questions  within  each  QMM  section  and  among  the  QMM  sections  is  required  to  focus 
the  QMM  survey  instrument  on  the  areas  which  are  the  most  important  in  determining 
and  measuring  the  quality  of  management  on  a  software-development  program.  Lastly, 
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reduce  the  total  number  of  questions  if  it  is  determined  that  this  is  appropriate.  Reducing 
the  number  of  questions  on  the  survey  instrument  includes  examining  tradeoffs  related  to 
the  usefulness  of  the  survey  instrument  versus  the  ease  of  executing  the  survey 
instrument  if  it  is  shorter  in  length. 

The  QMM  survey  instrument  requires  more  testing  to  formally  verify  and 
validate  it.  Unfortunately,  most  programs  do  not  keep  historical  data  which  can  be  used 
to  formally  validate  the  QMM  survey  instrument.  Data  appropriate  for  a  more  formal 
validation  of  the  QMM  survey  instrument  includes  schedule,  cost,  budget,  requirements 
management,  manpower,  risk  management,  people  management,  and  estimation/planning 
management  data  throughout  the  lifetime  of  a  program  or  at  least  during  the  time  period 
when  the  program  manager  evaluated  is  in  charge  of  the  program.  Formal  verification 
and  validation  will  require  a  long  term  commitment  from  the  program  and  program 
manager  to  be  evaluated.  The  researcher  will  need  to  collect  the  data  at  intervals  to 
ensure  the  data  is  not  lost  as  time  passes.  The  QMM  survey  instrument  should  not  be 
executed  near  the  beginning  of  the  data  collection  as  it  may  induce  a  bias  in  the  way  the 
program  manager  manages.  People  concentrate  their  efforts  in  areas  where  they  know 
they  will  be  evaluated  and  giving  the  survey  instrument  to  the  program  manager  early  in 
the  program  may  alter  their  management  style.  This  does  not  preclude  gathering 
management  quality  data  early  in  the  data  collection  but  a  different  survey  instrument  or 
technique  should  be  employed  which  will  not  induce  bias.  One  interviewee  thanked  the 
researcher  for  the  opportunity  to  participate  in  the  current  QMM  research  as  they  found 
the  QMM  survey  instrument  questions  to  be  a  good  reminder  of  all  they  should  be  doing 
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to  manage  the  program.  This  sentiment  was  echoed  by  a  number  of  interviewees  as  it 
reminded  them  of  what  they  were  not  doing  on  the  program. 

Provide  feedback  from  the  QMM  survey  instrument  to  the  evaluated  program 
managers  to  assist  them  in  improving  their  performance.  An  intent  of  the  QMM  survey 
instrument  is  to  apply  it  to  the  program  manager  at  the  beginning  of  or  at  some  point 
during  the  execution  of  the  program,  give  them  feedback  on  their  performance  in  each 
area,  and  then  apply  it  to  the  program  manager  later  in  the  program  execution,  or  the 
execution  of  another  program  they  are  managing,  to  measure  the  improvements  in 
management  quality  and  provide  further  feedback.  The  QMM  survey  instrument 
currently  does  not  provide  specific  feedback  guidance  for  the  program  managers.  In  fact, 
the  QMM  survey  instrument  measures  the  performance  of  the  program  manager  at  a  high 
level  and  only  provides  feedback  to  the  program  manager  to  the  level  of  their  score  in 
each  of  the  four  QMM  sections:  requirements  management,  estimation/planning 
management,  people  management,  and  risk  management.  There  is  currently  no  provision 
for  providing  more  specific  feedback  to  the  program  manager  other  than  in  the  area  of 
people  management  which  is  made  up  of  four  sub-sections:  human  resources, 
communication,  leadership  and  technical  competency.  Research  is  required  to  evaluate 
the  specific  questions  in  each  section  and  determine  which  course  or  courses  of  action 
should  be  recommended  for  the  program  manager  to  pursue  to  improve  their  performance 
in  the  area  depending  upon  their  responses  to  the  individual  questions.  The  feedback  can 
be  in  the  form  of  an  automated  program  which  analyzes  the  results.  However,  each 
manager  is  different  and  their  personal  style  and  program  situation  is  different.  A  better 
way  might  be  to  implement  an  automated  program  to  rank  the  quality  of  management  in 
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specific  areas  and  to  suggest  possible  courses  of  action  with  the  intent  of  providing  the 
information  to  a  management  trainer  or  mentor  who  would  then  work  with  the  program 
manager  to  improve  their  performance  in  those  areas.  In  the  case  where  the  program 
manager  does  not  have  access  to  a  mentor  or  management  trainer,  the  responses  of  the 
automated  program  could  be  combined  with  a  Myers- 
Briggs  personality  profile  of  the  program  manager  to  suggest  possible  courses  of  action 
for  the  program  manager  to  pursue  and  sources  of  information  to  help  the  program 
manager  understand  the  suggested  courses  of  action  and  the  implications  of  the  actions. 

The  QMM  survey  instrument  appears  to  detect  differences  in  perception  between 
the  program  manager  and  the  development  team  on  what  practices  and  procedures  the 
program  manager  believes  are  implemented  and  working  and  the  actual  state  of 
understanding  and  implementation  of  the  practices  and  procedures  on  the  development 
team  level.  The  program  manager  feedback  mechanism  described  above  could  include 
detection  of  areas  where  the  perceptions  of  the  program  manager  and  the  development 
team  differ  greatly  and  the  information  could  be  used  to  alert  the  program  manager  and/or 
the  program  manager’s  management  trainer  or  mentor  to  the  problem.  Knowledge  of 
areas  of  potential  misunderstandings  enables  the  program  manager  to  begin  working  on 
opening  up  the  communication  channels  to  better  guide  the  development  team’s  efforts 
and  receive  feedback  on  the  processes  and  procedures  which  the  program  manager 
implements,  enabling  them  to  improve  them. 

The  QMM  survey  instrument  results  could  be  used  as  an  input  into  current 

program  cost,  risk,  and  schedule  estimation  models  to  improve  the  resultant  estimations. 

Currently  these  models  do  not  incorporate  the  quality  of  software-development 
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management  as  a  factor  other  than  to  assume  good  management.  As  this  assumption  is 
not  necessarily  a  good  assumption,  using  the  QMM  results  input  for  the  models  may 
increase  the  accuracy  of  the  estimation  results.  Barry  Boehm  [Ref.  2]  states  the 
COCOMO  model  assumes  good  management  for  two  reasons:  the  quality  of  management 
is  difficult  to  measure,  and  providing  a  poor  manager  with  more  resources  to  complete  a 
program  than  a  good  manager  is  bad  management  practice.  Since  the  COCOMO  model 
assumes  good  management  and  provides  estimates  for  managers  based  upon  this 
assumption,  the  model  will  estimate  costs  and  schedules  for  good  managers.  This  gives 
the  poor  managers  the  same  guidelines  as  the  good  managers.  It  is  human  nature  to  live 
up  to  or  down  to  expectations.  If  the  model  provides  estimates  based  upon  bad 
management,  will  this  lower  the  expectations  and  goals  of  the  manager  and  produce 
programs  with  lower  success  levels?  It  is  an  interesting  question.  For  planning  purposes, 
accurate  estimates  are  necessary.  However,  what  will  the  cost  of  accurate  estimates  be  in 
the  case  of  poor  management,  assuming  accurate  estimates  are  desired?  The  research 
involves  implementing  a  QMM  survey  score  as  an  input  into  the  current  program 
estimation  models  to  provide  “accurate”  data  on  the  quality  of  the  program  management, 
such  as  with  a  formal  model  of  risk  assessment  in  a  software  program;  for  example,  see 
[Ref.  8]. 
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APPENDIX  A.  QUALITY  MANAGEMENT  METRIC  SURVEY 

FORMS  WITH  SCORING 


Quality  Management  Metric  survey  instrument  forms  template  with  scoring. 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): _ 

ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  LF  A  SEQUENTIAL  OR  WA  T ERF  ALL  APPRO  A  CH  IS  USED  FOR  DEVELOPMENT^  (Requirements  page  2  o[2j. 
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ir  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
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Estimation/Planning  Management  page  1  of  2  score 


Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
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Proeram  Name  _  — 


Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: _ _ _ _ _ 


co 


XL 

o 

a 

I 

.&l 

x: 

c/a 

s 

■s 

JD 

T3 


8 

4> 
C3 

E 

.  ,  S 

§m 
S|J3 


r 

a 

x 

o 

P 

,sj 

J3 

Vi 

s- 

0) 

*T3 

<3 


VO 


S 

C3 

Q 


(D 

s 

a 

Z 

B 

a 

ob 

s 

Oh 


69 


ir  choice  section  Four:  (Risk  Management  (RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
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Risk  Management  page  1  of  2  score 


Pair  choice  section  Four:  (Risk  Management  (RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template 

No.  Requirements  Management  Questionnaire _ 

_ 1  PM  chose  to  have  a  formal  requirements  list  _ 

_ 2  Requirements  recorded  in  some  way  _ 

_ 3  Written  requirements  were  part  of  some  formal  document _ 

_ 4  Written  requirements  were  informal _ 

_ 5  At  least  some  requirements  were  oral  only  _ 

_ 6  All  stakeholders  were  identified  _ 

_ 7  All  stakeholders  participated  in  the  requirements  extraction _ 

_ 8  Some  stakeholders  participated  in  the  requirements  extraction _ 

_ 9  Management  extracted  requirements,  no  stakeholder  involvement _ 

10  Management  passed  requirements  to  development  team _ 

1 1  Stakeholders  not  involved  in  Management  extraction,  but  approves _ 

12  Management  gets  inputs  from  stakeholders,  then  develops  requirments _ 

13  Developers  work  informally  with  users  to  arrive  at  requirements _ 

14|Same  as  13,  but  management  oversees  and  formalizes 


If  a  waterfall  or  sequential  development  strate 


All  requirements  complete  before  desiqn 


SB 


No  N/A 

0  I  0 

_A _ 0_ 

0  0 


_o _ o_ 

_o _ o_ 

2 _ 1_ 

0  0 


0  0 


Some  requirements  left  incomplete  prior  to  design 

-1 

0  0 

Requirements  informal  prior  to  design  effort 

-1 

0  0 

Requirements  serve  as  input 


Length  of  time  for  requirements  work  greater  than  development  work 


Requirements  developed  in  parallel  to  desiqn 


,  throwaway,  or  other  development  strate 


Learn  about  requirements  through  development  efforts 


No  coding  until  all  requirements  are  defined 


Requirements  formal  prior  to  desiqn  effort 


Requirements  serve  as  output 


Requirements  definition  work  in  parallel  to  development  efforts 


Requirements  developed  in  parallel  to  desiqn 


Are  requirements  frozen  at  some  phase 


Change  management  exists 


Change  management  is  formal 


'  is  consistent  throughout  development 


Requirements  are  updated 


Configuration  Management  (CM)  exists 


CM  is  formal 


Requirements  are  testable 


Requirements  testing  considered/implemented  during  extraction 


Requirements  testing  plan  exists 


Requirements  testing  is  formal 


All  requirements  have  priorities 


All  requirements  must  be  implemented 


Requirements  are  tested 


All  requirements  are  equally  important 


At  least  some  requirements  have  priorities 


All  requirements  are  traceable 


Traceability  not  important 


Each  requirement  has  an  author 


Who  authored  requirement  is  not  important 


Initial  set  of  requirements  to  be  implemented,  no  requirements  cree 


Structured  and  tracked  changes  to  requirements  onl 


Change  is  inevitable,  changes  allowed  at  all  times 


Change  is  inevitable,  but  changes  limited 


Requirements  control  fundin 


Requirements  history  kept 


Baseline  established  for  requirements  at  some  point  prior  to  develop 


TOTAL  SCORIN 

Enter  total  score  on  QMM  score  sheet  block  e. 

72 


Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template  Date. 

No.  Estimation/Planning  Questionnaire _ _ _ Yes  No  N/A 


1 

A  volume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

1 

0 

0 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

1 

0 

0 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

1 

0 

0 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

1 

0 

0 

5 

Product  metrics  tracked  and  updated  throughout  program  execution 

2 

-1 

0 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

1 

0 

0 

7 

Time  measure  process  metric  used  (cycle  time) 

1 

0 

0 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

2 

-1 

0 

9 

Proqram  cost  estimations  made  from  product  or  process  metrics 

1 

0 

0 

10 

Proqram  cost  estimations  tracked  and  updated  to  reflect  progress/changes 

1 

0 

0 

11 

Factor  analysis  performed  on  program 

1 

0 

0 

12 

Program’s  primary  purpose,  including  major  functions  and  deliverables  known 

2 

-1 

0 

13 

Work  breakdown  structure  developed 

2 

-1 

0 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

1 

-1 

0 

15 

Schedules  developed  based  on  realistic  expectations 

1 

-1 

0 

16 

Schedules  tracked  and  updated  based  on  new  information 

1 

-1 

0 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

1 

-1 

0 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

1 

-1 

0 

19 

COCOMO  estimates  performed 

1 

-1 

0 

20 

CSCI  clearly  defined  and  tasked 

2 

-1 

0 

21 

Estimates  completed  ad  hoc 

-2 

0 

0 

22 

Gantt  charts  used  and  updated 

1 

-1 

0 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

1 

-1 

0 

24 

Earned  value  established 

2 

-1 

0 

25 

Earned  value  tracked  throughout  program 

2 

0 

0 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

1 

-1 

0 

27 

Critical  path  for  proqram  tasks  developed  and  tracked 

2 

-1 

0 

28 

Meaure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

1 

0 

0 

29 

Estimates  are  updated  routinely 

2 

-1 

0 

30 

Schedules  are  updated  routinely 

2 

-1 

0 

31 

Estimations  are  made  by  program  management  (top-down) 

1 

o 

0 

32 

Estimations  are  made  by  program  team  members  (bottom-up) 

2 

0 

0 

33 

Automated  proqram  tracking  used 

1 

0 

0 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

1 

-1 

0 

35 

WBS  developed  only  as  data  call,  not  used  in  planning 

-1 

0 

0 

36 

Earned  value  used  to  track  program  progress 

2 

-1 

0 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

1 

-1 

0 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

2 

-1 

0 

39 

All  proqram  team  members  involved  in  planning  process 

2 

-1 

0 

40 

Hardware  also  considered  in  estimation  process 

1 

-1 

0 

41 

Proqram  history  compiled 

1 

0 

0 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

1 

-1 

0 

43 

Management  duties  apart  of  each  team  member’s  responsibilities 

-1 

1 

0 

44 

PM  dictates  schedules  to  program  team 

-1 

0 

0 

45 

Code  reviews  planned  in  schedule 

1 

-1 

0 

46 

Defined  tangible  milestones  established  for  program  tasks 

2 

-1 

0 

47 

Test  planninq  done  at  the  start  of  the  program 

1 

-1 

0 

48 

Estimations  are  completed  by  those  performing  the  tasks 

1 

-1 

0 

49 

Sensitivity  analysis  performed  for  program  choices 

1 

-1 

0 

50 

Software  deployment  planning  completed 

1 

-1 

0 

TOTAL  SCORING 

_ 1 

Enter  total  score  on  QMM  score  sheet  block  f. 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template  Date. 


No.  People  Management  Questionnaire _ Yes  No  N/A 


1 

PM  is  accessable  in  person  by  each  team  member 

1 

0 

0 

2 

PM  is  accessable  via  email  by  each  team  member 

1 

0 

0 

3 

PM  is  accessable  via  phone  by  each  team  member 

1 

0 

0 

4 

PM  not  only  considers  a  person’s  suitability,  not  also  desire  to  be  on  a  team 

1 

0 

0 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

1 

0 

0 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

2 

-1 

0 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

2 

-1 

0 

8 

PM  lets  teams  make  decisions  affecting  their  work 

1 

0 

0 

9 

PM  frequently  makes  decisions  without  any  consultation  with  members 

-2 

2 

0 

10 

PM  understands  the  technology/language  of  the  program 

1 

0 

0 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

1 

-1 

0 

12 

PM  prioritizes  problems  or  conflicts  within  the  program 

1 

0 

0 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

1 

-1 

0 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

1 

-1 

0 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

1 

-1 

0 

16 

PM  specifically  assigns  work  to  each  program  member 

1 

-1 

0 

17 

PM  sets  communication  protocol  to  be  followed 

1 

0 

0 

18 

PM  allows  unrestricted  communications 

1 

0 

0 

19 

PM  readily  makes  tough  decisions 

1 

-1 

0 

20 

PM  Jakes  control  in  difficult/  problem  areas 

1 

0 

0 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

1 

0 

0 

22 

PM  maintains  regular  communications  with  all  stakeholders 

2 

-1 

0 

23 

PM  maintains  regular  communications  with  users 

2 

-1 

0 

24 

PM  encourages  program  team  communication  with  users 

1 

-1 

0 

25 

PM  encourages  program  team  communication  with  stakeholders 

1 

-1 

0 

26 

PM  facilitates  horizontal  communication  within  program 

1 

-1 

0 

27 

PM  facilitates  communication  during  integration 

1 

-1 

0 

28 

PM  holds  meetings  without  clear  objectives  listed  prior  to  meeting 

-1 

2 

0 

29 

PM  must  approve  all  decisions  within  the  program 

-1 

1 

0 

30 

PM  must  approve  all  interactions  with  stakeholders 

-1 

1 

0 

31 

PM  must  approve  all  interactions  with  users 

-1 

1 

0 

32 

PM  makes  all  presentations  to  stakeholders/users 

0 

1 

0 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

1 

0 

0 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

1 

0 

0 

35 

PM  is  readily  willing  to  listen  to  program  problems  and  complaints 

1 

-1 

0 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

1 

-1 

0 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

1 

-1 

0 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

-1 

1 

0 

39 

PM  recruits  program  team  members  from  outside  organization 

1 

-1 

0 

40 

PM  directs  what  needs  to  be  done  and  directs  how  to  do  it 

-1 

1 

0 

41 

Program  personnel  have  clearly  defined  specific  tasks 

0 

1 

0 

42 

Although  individual’s  tasks  are  specific,  each  exposed  to  the  “bigger  picture" 

2 

-1 

0 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

2 

-1 

0 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

1  ' 

0 

0 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

2 

-1 

0 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

2 

-1 

0 

47 

PM  clearly  separates  technical  from  managerial  roles  for  individuals 

0 

1 

0 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

0 

1 

0 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

2 

-1 

0 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

2 

-1 

0 

TOTAL  SCORING 

_ 1 

Enter  total  score  on  QMM  score  sheet  block  g. 
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Program  Name _  YES-NO-N/A  Questionnaire  Scoring  Template  Date 


No.  Risk  Management  Questionnaire  Yes  No  N/A 

1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

4 

-4 

0 

2 

RM  is  formal  and  documented 

3 

-3 

0 

3 

A  specific  RM  plan  exists 

2 

-2 

0 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

-1 

1 

0 

5 

RM  is  done  prior  to  the  program  execution 

1 

0 

0 

6 

RM  is  done  by  an  outside  entity  to  the  development 

1 

0 

0 

7 

RM  is  done  internally  only 

0 

1 

0 

8 

RM  is  both  internally  performed  and  externally  assessed 

1 

-1 

0 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

1 

-1 

0 

10 

Risk  Assessment  is  only  a  management  function 

0 

1 

0 

11 

RM  is  informal  or  non  existent 

-1 

1 

0 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

1 

0 

0 

13 

Risks  are  only  generalized 

-1 

0 

0 

14 

Each  risk  is  delineated 

1 

0 

0 

15 

Each  risk  has  a  consequence 

1 

0 

0 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

1 

0 

0 

17 

Each  risk  has  a  mitigation  strategy 

1 

0 

0 

18 

Risk  Management  is  automated 

1 

0 

0 

19 

Risks  are  tracked 

2 

-2 

0 

21 

Regret  analysis  performed 

2 

0 

0 

22 

RM  drives  decisions  in  the  program 

3 

-2 

0 

23 

Risks  have  probabilities 

1 

0 

0 

24 

Risk  Management  is  ad  hoc 

-3 

0 

0 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

1 

0 

0 

26 

Risks  are  weighed  relative  to  other  program  risks 

1 

0 

0 

27 

Risk  Assessment  is  a  program  team  activity 

1 

0 

0 

28 

Risk  Assessment  done  prior  to  program  start 

2 

-1 

0 

29 

Risk  Assessment  includes  personnel  risk 

1 

-1 

0 

30 

RM  uses  tools,  but  depends  on  human  decisions 

2 

-1 

0 

31 

Risk  Assessment  includes  cost  risks 

1 

0 

0 

32 

Risk  Assessment  includes  schedule  risks 

1 

0 

0 

33 

Risk  Assessment  includes  technology  risks 

1 

-1 

0 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

1 

-1 

0 

35 

Risk  Assessment  includes  requirements  risks 

1 

-1 

0 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

1 

0 

0 

37 

Risk  Assessment  includes  documentation  risks 

1 

0 

0 

38 

Risk  Assessment  includes  integration  risks 

1 

-1 

0 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

1 

-1 

0 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

1 

-1 

0 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

1 

0 

0 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

1 

0 

0 

43 

High  risk  have  measured  tracking  (high  profile  status) 

1 

0 

0 

44 

Organizational  history  used  to  search  for  risks 

1 

0 

0 

45 

Other  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

46 

Internal  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

1 

0 

0 

48 

Risk  Assessment  includes  internal  organization  risks 

1 

0 

0 

49 

Risk  Assessment  includes  stakeholder  risks 

2 

-1 

0 

50 

No  risk  management  needed;  program  is  straightforward ed  &  understood 

-3 

3 

0 

TOTAL  SCORING 

1 

Enter  total  score  on  QMM  score  sheet  block  h. 
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Program: 


QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

1 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

i 

Score 

Requirements  Management 

a 

e 

0.00 

□ 

0.92 

K 

0.00 

Est./Planning  Management 

b 

t 

0.00 

B 

0.67 

__ 

0.00 

People  Management 

H 

g 

0.00 

1.86 

■ 

0.00 

Risk  Management 

d 

El 

0.00 

B 

0.55 

H 

0.00 

QMM  SCORE _ 0.00 

Max.  QMM  score  possible  528.00 

Min.  QMM  Score  possible  -130.86 

QMM  percentage  score  19.86% 


Objective/Subjective  view  of  the  overall  success  of  program  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Program: 

Success  Score: 
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APPENDIX  B.  RESEARCH  DATA 


This  appendix  contains  the  survey  responses  to  each  question  for  all  survey 
participants.  The  participants  are  identified  by  the  letter  of  the  program  with  a  dash 
followed  by  a  P  for  the  program  manager  and  a  number  for  the  individual  development 
team  members.  Each  table  has  the  corresponding  QMM  survey  section  title  at  the  top 
and  the  number  of  the  question  is  in  the  left  most  column.  Individual  responses  to  the 
questions  are  indicated  with  a  “1”  in  the  appropriate  box.  The  abbreviations  used  in  the 
tables  are  the  following: 

Lt.  -  indicates  the  answer  selected  is  the  left-hand  answer  on  the  pair-choice 
survey  row 

Rt.  -  indicates  the  answer  selected  is  the  right-hand  answer  on  the  pair-choice 
survey  row 

yes  -  indicates  the  answer  selected  is  the  yes  column  on  the  yes/no  survey 
no  -  indicates  the  answer  selected  is  the  no  column  on  the  yes/no  survey 
n/a  -  indicates  the  answer  selected  is  the  not  applicable  column  on  the  yes/no 
survey 
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Part  I  Pair-Choice:  Requirements  Management  (Page  1  of  2) _ _ 

I  A-P  |  Al  l  B-P  |  IM  |  C-P  I  CM  |  D-P 
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Part  I  Pair-Choice:  Estimation/Planning  Management  (Page  1  of 

I  A-P  I  /W  I  B-P  I  B-l  T  CP  !  C 
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Part  I  Pair-Choice;  Estimation/Planning  Mana 
I  f-p  I  F-i  I  g-p  I  CM 
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G-l  H-P  H-l 


le  Management  (Pa 
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B-l 


Part  I  Pair-Choice:  People  Management  (Page  1  of  2)  _ 

I  f-p  I  f-i  I  G-P  I  G-i  I  iTp  1  H^l 
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Part  I  Pair-Choice;  Risk  Management  (Pag 
I  A-P  I  aH  I  bp  I  B-l 
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Part  I  Pair-Choice:  Risk  Management  (Page  1  of  2) 
I  I  F-l  I  g-p  |  (M  | 
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H-P 
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Part  II  Yes/No:  Estimation/Planning  Management 
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APPENDIX  C.  BAR  CHARTS  OF  THE  RESEARCH  DATA 


This  appendix  contains  bar  charts  of  the  research  data. 
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Program  Level  of  Success 


Program  Managi 


Figure  Cl 


Program  Level  of  Success 
0  =  Total  Failure 
10  =  Total  Success 


Program  Level  of  Success 
0  =  Total  Failure 
10  =  Total  Success 


PM  and  IND  Program  and  QMM  Scores 


Program 


Figure  C3.  Program  Manager  and  Individual  Development  Team  Member  Overall 
Program  Scores  and  QMM  Percentage  Scores 
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Program  Level  of  Success 
0  *s  Total  Failure 
10  =  Total  Success 


Program 


Figure  C5.  Program  Manager  and  Individual  Development  Team  Member  QMM 

Percentage  Scores 


Possible  Score 
)f  Possible  Score 


Requirements  Management  Scores 


Program 


Figure  C6.  Program  Manager  and  Individual  Development  Team  Member 
Requirements  Management  Scores 
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0  =  0%  of  Possible  Score 
10  =  100%  of  Possible  Score 


Estimation/Planning  Scores 


Program 


Figure  Cl.  Program  Manager  and  Individual  Development  Team  Member 
Estimation/Planning  Management  Scores 
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0  =  0%  of  Possible  Score 
10s  100%  of  Possible  Score 


People  Management  Scores 


Figure  C8.  Program  Manager  and  Individual  Development  Team  Member  People 

Management  Scores 
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Score 

0  =  0%  of  Possible  Score 
10  =  100%  of  Possible  Score 


Risk  Management  Scores 


Program 


Figure  C9  Program  Manager  and  Individual  Development  Team  Member  Risk 

Management  Scores 
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